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ABSTRACT 


This  thesis  develops  statistical  analysis  in  support  of  Readiness-Based  Sparing 
(RBS)  for  U.S.  Navy  aviation  weapon  systems.  RBS  seeks  to  determine  the  least-cost 
allowance  list  to  meet  pre-specified  operational  availability  of  specifically  identified 
systems.  The  research  shows  how  RBS  products  such  as  the  Navy  Aviation  RBS  Model 
(NAY ARM)  can  be  used  by  leadership  and  builders  to  anticipate  changes  in  RBS  cost  as 
a  function  of  changes  in  key  inputs.  We  develop  NAY  ARM  Experimental  Designs 
(NED),  a  computational  tool  created  by  applying  a  state-of-the-art  experimental  design  to 
the  NAVARM  model.  Statistical  analysis  of  the  resulting  data  identifies  the  most 
influential  cost  factors.  Those  are,  in  order  of  importance,  availability  goal,  unit  price, 
wartime  flying  hours,  maintenance  rate  to  failure,  high  priority  order  and  ship  time, 
number  of  aircraft,  wholesale  delay  time,  ratable  pool  factor,  intermediate  maintenance 
activity  repair  time,  and  mean  time  to  repair.  Seventy-five  percent  of  NED  predictions  are 
within  a  3%  or  less  error  of  actual  values  for  changes  within  ±10%  to  baseline  values, 
and  all  predictions  are  within  7%. 
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EXECUTIVE  SUMMARY 


The  Naval  Supply  Systems  Command  Weapons  System  Support  (NAVSUP 
WSS)  Office  Code  N421  establishes  inventory  levels  for  thousands  of  items  to  ensure 
readiness  of  aviation  weapon  systems.  Since  1985,  Readiness-Based  Sparing  (RBS)  is  the 
concept  and  mandated  method  to  set  these  aviation  weapon-system  inventory  levels. 
(Naval  Inventory  Control  Point,  2008,  p.  4)  RBS  models  seek  pre-specified  levels  of 
operational  availability  (Ao)  for  multiple  weapon  systems  at  minimum  cost.  There  are 
several  RBS  models  and  tools  available  to  NAVSUP  WSS.  However,  NAVSUP  WSS 
cannot  assess  the  sensitivity  of  the  solution  (specifically  cost),  other  than  modifying  the 
key  inputs  and  running  each  individual  instance. 

In  2016,  faculty  at  the  Naval  Postgraduate  School  developed  the  Navy  Aviation 
RBS  Model  (NAVARM),  a  heuristic  optimization  model  for  single-site  and  multi- 
indentured  RBS  problems.  (Salmeron,  2016)  NAVSUP  WSS  code  N421  suggested  NPS 
conduct  a  formal  study  of  influential  factors  that  affect  RBS  costs  calculated  by 
NAVARM.  Since  NAVARM  is  open  source,  we  develop  the  NAVARM  Experimental 
Designs  (NED)  tool  to  assess  the  influential  factors. 

The  thesis  objective  is  to  identify  the  factors  most  sensitive  to  the  NAVARM 
output  and  find  the  meta-models  that  estimate  RBS  cost  with  minimal  error.  To  enhance 
this  study,  N421  provides  us  with  ten  test  cases  that  we  can  use  to  make  our  assessments. 
The  test  cases  vary  across  multiple  aviation  platforms  on  both  coasts.  Examples  of  these 
platforms  are  USS  Harry  S.  Truman  (CVN  75)  in  Norfolk,  Virginia  and  Marine  Aviation 
Logistics  Squadron  1 1  in  San  Diego,  California. 

We  integrate  a  nearly  orthogonal  and  nearly  balanced  (NOB)  mixed  design 
spreadsheet  with  NAVARM.  (Vieira,  2012)  NOB  provides  designs  that  are  “low 
maximum  absolute  pairwise  correlation  and  imbalance,”  thereby  constructing  fully 
spread-out  and  equally  balanced  values.  (Vieira  et  al.,  2013,  p.  273)  NOB  is  known  to 
improve  the  cost  estimate  precision  with  less  variance.  We  generate  a  ±10%  scaling 
value  in  the  NOB  spreadsheet  and  apply  it  to  the  baseline  values  of  the  following  13 
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factors  to  all  test  cases:  expanded  war  flying  hours;  quantity  per  application;  intermediate 
maintenance  activity  repair  time;  high  priority  order  and  ship  time;  wholesale  delay  time; 
unit  price;  maintenance  rate  to  failure;  ratable  pooling  factor;  flying  hours;  mean  time  to 
repair;  number  of  aircraft;  RBS  performance  goal;  and  wartime  flying  hours. 

Since  NAVARM  operates  in  Visual  Basic  for  Applications  (VBA),  we  develop  a 
set  of  VBA  subroutines  that  interact  with  the  NAVARM  model.  This  process  also 
captures  the  simultaneous  variations  of  the  13  factors  listed  above  and  merges  them  with 
NAVARM  RBS  cost.  We  expect  that  this  design  of  experiments  will  identify  the 
relationship  between  factors  and  the  NAVARM  RBS  cost. 

After  paring  the  data  from  multiple  trials,  we  perform  a  stepwise  regression  using 
the  statistical  software.  We  identify  the  most  impactful  factors  along  with  the  best  meta¬ 
model  for  estimating  NAVARM  RBS  cost  for  each  test  case.  In  order  of  importance,  the 
factors  are  availability  goal,  unit  price,  wartime  flying  hours,  maintenance  rate  to  failure, 
high  priority  order  and  ship  time,  number  of  aircraft,  wholesale  delay  time,  ratable  pool 
factor,  intermediate  maintenance  activity  repair  time,  and  mean  time  to  repair.  Major 
sensitivity  assessments  are  as  follows: 

1.  Meta-model  development  using  stepwise  regression  indicates  that  60%  of  the 
models  have  only  main  effects  (no  two-way  interactions  or  quadratic  effects). 

2.  Four  test  candidate  files  have  a  quadratic  effect.  The  test  candidate  files  with 
the  quadratic  effect  are  USS  Bataan  (LHD  5),  USS  BonHomme  Richard 
(LHD  6),  USS  Iwo  Jima  (LHD  7),  and  FMS  Denmark.  Although  these  test 
candidate  files  are  for  sites  with  rotary  wing  aircraft  parts,  we  cannot  conclude 
that  rotary  wing  aircraft  cause  this  effect. 

3.  Exponential  and  reciprocal  transformations  of  one  factor,  availability  goal, 
show  no  improvement  to  the  overall  meta-model  development  for  those 
factors  with  non-linearity.  Both  transformations  on  availability  goal  cause  R- 
Square  adjusted  to  decrease,  Root  Mean  Square  Error  to  increase,  F  ratio  to 
decrease,  and  t  Ratio  to  decrease  compared  to  the  non-transformed  meta- 
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models.  This  indicates  that  the  quadratic  fits  best  among  the  choices  of 
transforming  availability  goal,  vice  exponentially  or  reciprocally. 

4.  One  of  the  test  candidate  files.  Naval  Air  Facility  Misawa,  has  main  effects, 
no  quadratic  effects,  and  one  two-way  interaction. 

5.  Both  USS  Bataan  (LHD  5)  and  USS  Iwo  Jima  (LHD  7)  test  candidate  files 
have  main  effects,  one  quadratic  effect,  and  one  two-way  interaction. 

6.  The  NED  meta-model  predictions  have  50%  of  their  predictions  within  a 
0.05%  to  2%  error  range  for  the  USS  Harry  S.  Truman  (CVN  75)  test 
candidate  file.  The  results  of  the  other  nine  test  candidate  files  have  nearly 
75%  of  their  predictions  within  a  3%  or  less  error,  while  predicting 
NAVARM  RBS  cost.  NED  allows  the  user  to  make  estimations  of  cost  for  all 
test  cases  within  7%  of  actual. 

All  test  cases  except  Maritime  Aviation  Logistic  Squadron  1 1  (MAL)  have  either 
goal  or  unit  price  as  their  number  one  factor.  The  MAL  test  case  has  wartime  flying 
hours  as  its  number  one  factor  with  unit  price  as  second  and  goal  as  its  third.  The  fact 
that  Marine  Corps  is  operating  with  less  than  half  its  aircraft  available  suggests  that  the 
remaining  aircraft  are  being  overused,  resulting  in  greater  wear  and  tear  and  yielding 
reduced  airworthiness.  Since  this  is  based  on  retrospective  data  we  cannot  establish 
causality,  but  further  investigation  is  warranted. 

Overall,  we  take  a  prognostic  approach  to  conducting  this  research.  We  develop 
NED  to  make  predictions  from  data  generated  by  running  thousands  of  NAVARM 
simulation  trials  over  ten  different  aviation  locations  and  platforms.  This  research  furthers 
the  development  of  the  desired  tool  for  NAVSUP  WSS  Office  Code  N421.  N421  can 
now  use  current  prediction  expressions  for  the  ten  given  cases  when  the  changes  to  the 
existing  factors  are  within  ±10%.  If  the  changes  exceed  ±10%,  we  can  use  NED  with 
the  new  NOB,  and  analyze  the  output  with  any  statistical  software  that  includes  stepwise 
regression  for  updated  prediction  expressions.  However,  in  its  current  format,  NED 
cannot  accommodate  new  test  cases  and/or  new  factors. 
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I.  INTRODUCTION 


There  ain’t  no  rules  around  here!  We’re  trying  to  accomplish  something! 


— Thomas  Edison, 
American  inventor 


The  Naval  Supply  Systems  Command  Weapons  System  Support  (NAVSUP 
WSS)  mission  “is  to  provide  Navy,  Marine  Corps,  Joint  and  Allied  Forces  program  and 
supply  support  for  the  weapons  systems  that  keep  our  naval  forces  mission 
ready”  (NAVSUP  WSS.,  2017,  Mission,  para.  1).  The  primary  focus  of  NAVSUP  WSS 
Philadelphia  is  on  weapons  system  and  aviation  support  through  Readiness-Based 
Sparing  (RBS).  RBS  models  seek  to  determine  the  least-cost  allowancing  (i.e., 
establishment  of  inventory  levels)  to  meet  pre-specified  operational  availability  (Ao)  for 
all  Weapon  Systems  (WS).  Each  of  these  WS  consists  of  multi-indentured  parts  in  the 
range  of  tens  of  thousands.  The  Department  of  Defense  has  used  a  number  of  RBS 
models  since  the  1960s  (Defense  Acquisition  University,  2012).  These  models  include 
the  Aviation  Readiness  Requirements  Oriented  to  Weapon  Replaceable  Assemblies 
(ARROWS),  the  Service  Planning  Optimization  (SPO)  models,  and  Repairable  Integrated 
Model  for  Aviation  (RIMAIR).  (Note:  ARROWS,  SPO,  and  RIMAIR  are  not  available  to 
the  researcher,  and  are  only  discussed  for  informational  purposes.)  Naval  Postgraduate 
School  faculty  and  students  are  developing  the  Navy  Aviation  RBS  Model  (NAVARM) 
to  guide  NAVSUP  WSS  allowance  setting. 

An  RBS  model  consists  of  multiple  key  inputs  such  as:  rotable  pool  factor  (RPF), 
wartime  flying  hours  (WFHRS),  Aogoal,  Unit  Price,  high  priority  order  and  ship  time 
(HPOST),  low  priority  order  and  ship  time  (LPOST),  wholesale  delay  time  (WDT), 
intermediate  maintenance  activity  repair  time  (IMARPT),  maintenance  rate  to  failure 
(MRF),  expanded  war  flying  hours  (EXPWFHRS),  quantity  per  application  (QPA), 
number  of  aircraft  (NUMWS),  mean  time  to  repair  (MTTR),  and  flying  hours  (FHRS). 
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These  inputs  are  used  to  acquire  Aviation  Consolidated  Allowance  List  (AVCAL) 
packages.  Input  values  will  vary  by  the  type  of  allowance  package,  operational  necessity, 
and  supported  aircraft.  The  input  values  are  originated  by  Navy  Enterprise  Resource 
Planning,  NAVSUP  WSS  internal  business  rules,  and  fleet  maintenance,  as  well  as  policy 
from  the  Office  of  the  Chief  of  Naval  Operations  (OPNAV)  (Sax,  2012,  pp.  4-7).  As  a 
result,  NAVSUP  WSS  can  improve  efficiency  and  resource  allocation  by  enriching  the 
understanding  of  how  these  multiple  inputs  affect  cost.  Prior  work  on  RBS  assessment 
has  involved  determining  the  factor  influence  of  the  ARROWS  model  to  determine  RBS 
cost  by  varying  one  input  at  a  time.  The  impact  of  jointly  varying  inputs  has  never  been 
previously  assessed.  This  thesis  develops,  and  computationally  implements,  NAVARM 
Experimental  Designs  (NED)  in  order  to  provide  insight  into  the  question,  “What  are 
NAVARM  RBS  cost’s  most  influential  factors?” 

A.  PROBLEM  INTRODUCTION 

In  February  2017,  Defense  News  reported  that  nearly  two-thirds  of  the  U.S. 
Navy’s  F/A-18  Hornet  and  Super  Hornets  were  grounded  due  to  a  shortage  of  parts  at 
aviation  depot  level.  (Cavas,  2017)  The  article  also  stated  that  53%  of  all  of  the  Navy’s 
aircraft  were  grounded  as  a  result  of  Continuing  Resolution  Authority  budget  constraints, 
maintenance  issues,  and  long  lead  times  for  spare  parts.  A  recent  example  of  this 
problem,  as  reported  in  February  of  2017,  was  a  reduction  in  mission  capable  spare  parts 
available  to  the  Marine  Corps,  which  resulted  in  only  439  of  their  1,065  aircraft  to  be 
airworthy.  The  Marine  Corps  had  to  reduce  the  number  of  MV-22  Ospreys  from  twelve 
aircraft  to  six  in  Africa  due  to  their  inability  to  sustain  them  in  the  crisis  response  task 
force  (Seek,  2017). 

Currently,  the  Operations  Analyst  Office  Code  N421  at  NAVSUP  WSS  in 
Philadelphia,  PA,  uses  “Readiness  Suite”  to  create  an  AVCAL.  Readiness  Suite  is  a 
computer  system  that  combines  many  tools  into  a  central  location,  including  SPO, 
RIMAIR  and  ARROWS  (Sax,  2012,  p.  2).  In  creating  AVCALs  most  of  the  work  is 
centered  on  using  the  SPO  software,  a  commercial,  off-the-shelf  product.  For  the  purpose 
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of  this  research,  SPO  and  ARROWS  are  not  used  to  analyze  key  factors  contributing  to 
RBS  output. 

NAVSUP  WSS  Office  Code  N421  wishes  to  have  a  stand-alone  organic  system 
like  NAVARM  that  will  provide  them  with  more  flexibility  in  building  AVCALs  for 
different  platforms  and  sites,  and  that  can  be  adjusted  easily  for  various  Weapon  Systems 
(WS).  Even  with  a  tool  like  NAVARM,  the  N421  team,  to  some  extent,  is  unsure  about 
how  cost  is  influenced  by  the  previously  mentioned  factors  (Huff,  personal 
communication,  July  12,  2017). 

B.  SCOPE 

This  thesis  will  identify  the  factors  that  have  the  greatest  impact  on  NAVARM 
RBS  cost.  Through  design  of  experiments  (DOE),  we  develop  meta-models  that  predict 
the  total  AVCAL  cost  for  various  aviation  sites  located  ashore  and  at  sea.  The  research 
will  use  NAVARM  version  1.31.  It  will  identify  NAVARM  RBS  output  (RBS  cost)  by 
varying  a  combination  of  factors.  Separate  analyses  are  performed  by  site  location. 

This  research  is  expected  to  help  reduce  the  N421  production  run  and  analysis 
time  by  an  amount  between  two  and  fifteen  hours  per  week.  The  research  will  afford 
N421  the  opportunity  to  better  serve  allowance  builders  in  building  AVCALs,  and 
answer  data  calls  concerning  NAVSUP  WSS  budget. 

In  addition,  the  NED  tool  is  developed  and  implemented  in  an  environment  that 
allows  N421  the  opportunity  to  replicate  the  analyses  presented  in  this  thesis  as  well  as 
conducting  new  experimentation  by  varying  the  previously  mentioned  factors.  However, 
as  currently  implemented,  NED  does  not  allow  the  addition  of  new  factors  or  test  cases 
from  those  presented  in  this  study. 

C.  THESIS  OUTLINE 

The  four  remaining  chapters  of  this  thesis  are  organized  as  follows:  Chapter  II 
explores  the  history  and  background  of  RBS  and  acknowledges  previous  research 
completed  by  personnel  who  work  for  NAVSUP  WSS  Office  Code  N421.  Chapter  III 
provides  the  methodology  required  to  create  the  DOE  as  well  as  the  importance  and 
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reasoning  behind  the  Sensitivity  Analysis  (SA)  technique.  Chapter  IV  explores  the  results 
of  the  SA  and  Regression  analysis  conducted  from  the  DOE  simulated  trials.  Chapter  V 
provides  conclusions,  future  work,  and  recommendations. 
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II.  BACKGROUND 


The  difficulty  lies  not  so  much  in  developing  new  ideas  as  in  escaping 
from  old  ones. 

— John  Maynard  Keynes, 
British  economist 


This  chapter  will  expound  on  the  RBS  history  and  its  significance  within  the  U.S. 
Navy.  It  will  present  a  theoretical  view  of  the  NAVARM  RBS  solution,  and  the  SA 
accomplished  by  using  the  ARROWS  model. 

A.  LITERATURE  OVERVIEW 

Every  military  service  is  in  dire  need  to  improve  system  efficiency,  reduce  costs, 
and  keep  fleet  assets  like  aircraft  Fully  Mission  Capable  (FMC).  A  quick  overview  of 
history  will  show  that  the  RBS  approach,  both  in  concept  and  in  practice,  can  assist  the 
services  in  achieving  that  goal.  The  inventory  models  that  use  the  RBS  concept  are  not 
the  only  models  in  the  U.S.  military,  but  the  RBS  concept  is  one  that  supports  all  service 
branches. 


1.  Air  Force  Base  Field  Testing  of  Inventory  Model  for  Repairable  Items 

While  Sherbrooke  (2004,  p.  60)  was  working  for  the  RAND  Corporation  during 
the  1960s,  he  developed  and  implemented  an  inventory  model  for  the  Air  Force  known  as 
the  VARI-METRIC  model.  This  concept  is  the  basis  of  the  ARROWS,  SPO,  and 
NAVARM  RBS  approaches  to  establish  inventory  levels.  The  concept  develops  an 
approach  to  measure  performance  of  supplying  parts  by  measuring  backorders  instead  of 
fill  rate.  Fill  rate  is  a  percent  measure  of  demands  met  as  orders  are  placed  (Sherbrooke, 
2004,  p.  11).  For  the  remainder  of  this  thesis  we  use  the  terms  “RBS  approach,”  “RBS 
model,”  or  simply  “RBS”  to  refer  to  VARI-METRIC  concept.  Sherbrooke  initially  tested 
his  model  at  Hamilton  Air  Force  Base  (AFB).  With  the  help  of  computer  simulations,  he 
field-tested  one  tactical  aircraft  type,  which  resulted  in  an  increased  fill  rate  from  82.8% 
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to  91.2%,  while  reducing  total  investment  cost  from  $1.84M  to  $1.45M.  Even  more 
significantly,  the  aircraft  reduced  its  nonoperational  rate  by  42%  (Sherbrooke,  2004,  p. 
10).  Despite  this  promising  result,  the  VARI-METRIC  model  was  initially  criticized 
because  only  one  aircraft  type  was  tested  (Sherbrooke,  2004,  p.  10).  The  Air  Force  then 
conducted  a  major  test  of  the  model  at  George  AFB,  which  included  three  major  aircraft, 
the  F-4C,  F-104,  and  F-106,  during  two  six-month  periods  (Sherbrooke,  2004,  p.  10). 

The  first  six-month  period  was  the  “pretest”  period.  During  this  period,  the  Air 
Force  developed  a  baseline  with  its  current  model  to  compare  with  the  field-testing 
results  of  the  RBS  model.  The  field  testing  occurred  from  March  1,  1965,  until  August 
31,  1966.  During  both  the  pretest  and  field  testing  period,  three  aircraft  types  along  with 
3,673  repairable  items  were  evaluated,  and  the  results  were  outstanding.  As  presented  in 
Figure  1,  the  RBS  model  improved  performance,  and  reduced  the  investment  (budget)  by 
nearly  half.  Sherbrooke  and  his  team  also  noted  that  a  reduction  in  Special  levels  (seen  in 
Figure  1)  from  167  to  28  was  not  appropriate  for  the  Air  Force  to  achieve  large 
reductions  in  stock  levels.  They  also  noted  that  had  improvements  been  under  10%,  they 
would  have  dismissed  the  overall  test,  but  it  is  clearly  seen  from  the  summary  results 
presented  in  Figure  1  that  this  is  not  the  case  (Sherbrooke,  2004,  p.  11). 


Pretest 

Test 

%  Change  from  Pretest 

Investment  ($M) 

13.4 

7.3 

-46 

Fill  rate  (%) 

75 

76 

1 

Special  levels 

167 

28 

-83 

Aircraft  possessed 

114 

96 

-16 

Flying  hours/month 

3621 

2264 

-37 

Sorties/month 

2009 

1362 

-32 

Average  #  of 

71 

40 

-44 

backorders 

Figure  1.  George  AFB  test  results  during  Sept.  1,  1965-Aug.  31,  1966. 

Source:  Sherbrooke  (2004). 
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2.  RBS  Implementation  into  Naval  Aviation 

The  RBS  inventory  model  was  first  implemented  and  tested  for  the  Air  Force  in 
1966.  The  Navy  did  not  implement  the  RBS  model  until  the  mid-1980s.  The  Chief  of 
Naval  Operations  directed  the  Navy  Supply  Systems  Command  to  implement  RBS,  and 
directed  aviation  supply  to  embrace  the  concept  in  1985  (Naval  Inventory  Control  Point, 
2008,  p.  4).  RBS  was  first  used  to  develop  pack-up  kits  for  the  SH-60B  light  airborne 
multipurpose  system,  a  program  used  by  the  U.S.  Navy  for  anti-submarine  warfare. 
(House,  2000,  p.  46)  Later,  the  Operational  Analysis  Department  in  Mechanicsburg,  PA, 
was  tasked  with  the  development  and  implementation  of  the  RBS  model  to  create 
AVCALs  for  all  aviation  platforms.  The  resulting  model  is  known  as  ARROWS. 

ARROWS  testing  was  accomplished  by  comparing  model  predictions  with  the 
actual  inventory  from  the  Aviation  Supply  Office  for  the  SH-60B  and  F14A  during  the 
USS  Enterprise  deployment  of  1986  (Strauch,  1986,  p.  ii).  The  ARROWS  model  results 
were  compared  to  the  Navy’s  current  model,  (called  the  Availability  Centered  Inventory 
Model  (ACIM))  and  their  findings  revealed  that  the  ARROWS  model  maintained  a  high 
level  of  FMC  aircraft,  reduced  AVCAL  package  cost,  and  improved  overall  Ao  (Strauch, 
1986,  p.  ii).  The  analysis  team’s  recommendation  was  to  replace  the  ACIM  with 
ARROWS,  and  to  start  using  RBS  for  future  at  sea  testing.  ARROWS  would  become  the 
Navy  RBS  approach  for  aircraft  inventory  support  (Strauch,  1986,  p.  26). 

In  1993,  the  U.S.  Navy  was  able  to  fully  integrate  the  RBS  concept  on  board  the 
USS  America  (CV-66)  with  the  RBS  AVCAL.  This  initiative  and  analysis  reduced  the 
traditional  AVCAL  dollar  investment  by  $33  million.  This  was  accomplished  by 
increasing  the  cheaper  weapons  replaceable  units  National  Item  Identification  Number 
(NUN)  range  by  24%  while  decreasing  the  expensive  weapons  replaceable  units  NUN 
allowance  quantity  (House,  2000,  p.  46). 

3.  Readiness  Suite 

ARROWS  continued  to  dominate  as  the  Navy’s  RBS  model  throughout  the 
1990s,  as  desktop  computers  improved  in  computing  power.  The  overall  structure  of  the 
ARROWS  modeling  system  migrated  from  a  DOS  version  to  a  Windows-based  operating 
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system  (Sax,  2012,  p.  1).  Along  with  ARROWS,  the  Navy  had  a  multitude  of  demand- 
based  models  and  simulators.  Instead  of  cluttering  analyst  desktops  with  a  slew  of  tools, 
the  Navy  developed  the  Readiness  Suite  in  2005.  This  suite  included  the  web-based 
Naval  Online  Allowance  Handling  (NOAH)  system,  which  improved  effectiveness  of 
inputting  data,  standardized  business  rules,  automated  data  management,  and  allowed 
availability  of  multiple  tools  to  over  900  users  in  the  Navy  organization.  (Sax,  2012,  p.  1) 

As  more  RBS  concepts  evolved  and  multiple  tools  became  available  to  the 
analyst,  OPNAV  authorized  ARROWS,  SPO,  ACIM,  and  other  models  to  be  included 
in  the  Readiness  Suite,  which  is  depicted  in  Figure  2  (Chief  of  Naval  Operations,  201 1,  p. 
8.  Figure  2  shows  more  tools  and  options  available  through  the  Readiness  Suite  than  we 

will  discuss.  For  the  purpose  of  this  research,  our  interest  is  primarily  with  the  RBS 
concept  for  aviation,  and  those  models  that  are  used  to  plan  for  allowancing.  We  bring  to 
the  reader’s  attention  the  plethora  of  tools  the  analyst  has  available  at  NAVSUP  WSS. 


NAVY  ERP 


ReMAD 


READINESS  SUITE 


CRCS/CADS 

•  Usage  rates 

•  Configuration  data 

•  Workflow 

•  Data  Management 


Candidate  files. 
Wholesale  inputs 


Aviation 
Allowance 
Products 
,(X05,  X06,  etc.) 


NOAH 


M-NOAH 


Maritime  Allowance 

Products 

(TLL/OSISL/FILL) 


MCRCS/MADS 

•  Usage  rates 

•  Configuration  data 

•  Workflow 

Data  Management 


Modules  existing  in  RS  2007.1.1 

Modules  integrated  or  being  integrated  into  RS 

since  RS  2007.1.1 


SIMULATION  MODELS 


TIGER/SPO 


D-SCORE 


ALLOWANCE  MODELS 


PD68  1 1  DACM  (PD66) 


|  NECE  (Phase  1)  ||  OSISL/TLL  ~ 


MARITIME  ANALYSIS  TOOLS 


CART** 


]  AC  AT  I  RIC  only) 


FMS  (M)  Requirements 


ACR  and  PPR  Management 


**  Portions  migrated  from  ReMAD 
NOTE:  RS  supported  by  DB,  application,  and  web  servers 
located  in  NAHI 


Note:  The  tools  in  Readiness  Suite  are  not  available  to  the  researcher,  and  are  only 
mentioned  for  informational  purposes. 

Figure  2.  Readiness  suite  components  and  interactions.  Source:  Sax  (2012). 


8 


It  is  also  worth  noting  that  tools  like  SPO  are  commercial,  off-the-shelf  software 
that  will  be  used  in  conjunction  with  other  tools  like  ARROWS,  TIGER  (a  tool  similar  to 
ARROWS  but  used  for  maritime  WS),  and  ACIM.  Sax  states  in  the  paper  titled,  Aviation 
Allowancing  RBS  Overview,  that  SPO  is  a  “Flexible  model  used  to  compute  Site  Demand 
Based  Levels  (SDBLs),  Quarterly  Wholesale  Levels,  Adhoc  (Delta)  Wholesale  Levels, 
and  Readiness  Based  (RBS)  Allowances  for  AVCALs  and  large  SHORCALs  [Shore- 
based  Consolidated  Allowance  Lists]”  (Sax,  2012,  p.  3).  The  pictorial  layout  of  the  suite 
shows  that  experimental  designs  could  be  difficult  to  investigate  (Huff,  personal 
communication,  July  12,  2017). 

B.  THEORETICAL  FRAMEWORK 

When  the  Navy  adopted  the  RBS  approach,  it  developed  mathematical 
formulations  to  calculate  the  required  spares  for  aircraft  AVCALs  and  SHORCALs.  This 
section  explains  the  RBS  theory  behind  the  NAVARM  model. 

1.  RBS  Modeling  Calculations 

Before  the  basic  RBS  model  calculations  are  examined  in  detail,  the  RBS 
objective  needs  to  be  discussed.  According  to  OPNAV  Instruction  4441.5A,  the  RBS 
concept  is  a  methodology  for 

spares  and  repair  parts  allowance  determination  to  ensure  that  prescribed 
readiness  thresholds  and  objectives  are  achieved  at  the  lowest  possible 
cost.  Readiness  thresholds  are  expressed  as  either  operational  availability 
(Ao)  or  full  mission  capable  (FMC)  and  or  mission  capable  (MC)  rates. 

The  term  “RBS”  applies  to  single  echelon  and  single  indenture  systems,  as 
well  as  their  multi-echelon  (ME)  and  multi-indenture  (MI)  extensions. 

(Chief  of  Naval  Operations,  2011,  p.  1) 

Sherbrooke  outlines  the  following  assumptions  for  the  VARI-METRIC  theory 
used  for  RBS: 

•  All  locations  and  NIINs  follow  a  (s-1,  s)  inventory  policy,  where  s  (the 
inventory  position)  is  the  largest  stock  level  determined  from  a  location. 
When  an  order  is  placed  inventory  position  is  reduced  by  one  to  meet  the 
demand,  which  triggers  a  reorder.  Thus,  the  reorder  point  is  s-1.  An  order 
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quantity  of  one  is  justified  by  the  fact  that  the  NIINs  considered  are  high 
cost  and  low  demand. 

•  The  expected  backorders  (EBO)  by  location  are  calculated  based  on  a 
Poisson  assumption  for  the  rate  of  the  average  pipeline  for  each  NUN. 

•  In  theory,  the  overall  inventory  position  s  is  the  number  of  NIINs  on-hand 
plus  the  order  quantity  minus  the  EBOs. 

•  When  a  NUN  is  not  repairable  then  a  new  one  is  ordered  to  resupply  the 
location.  Also,  when  the  order  quantity  equals  one  the  inventory  position 
is  constant.  (Sherbrooke,  2004,  pp.  24-25) 

The  following  sub-sections  describe  the  RBS  process  in  sequence. 


a.  Average  ( Resupply  and  Repair)  Pipeline  Calculation 


The  RBS  model  will  calculate  the  average  pipeline  for  both  the  resupplying  and 
the  repairing  materiel  required  to  keep  all  fleet  assets  mission  capable.  These  calculations 
are  presented  in  Equations  (1)  and  (2): 


Resupply  Pipeline  = 

f 

Repair  Pipeline  = 

V 


MRF  x  QPA  x  NUMWS  x  WFHRS  x  HPOST 

\  90 

RPF  x  QPA  x  NUMWS  x  WFHRS  x  IMARPT  "j 


\ 

) 


(1) 

(2) 


where: 


MRF  ~  maintenance  rate  to  failure  (number  of  part  failures  per  100  flying  hours 
that  are  sent  to  depot  for  repair); 

QPA  ~  quantity  per  application  (number  of  a  particular  part  per  aircraft); 

WFHRS  ~  wartime  flying  hours  (number  of  flying  hours  a  squadron  fly  per 
quarter  divided  by  100); 

HPOST  ~  high  priority  order  and  ship  time  (number  of  days  to  transport  a  part 
from  the  stock  point  to  the  end  user  when  an  MRF  failure  occurs); 

NUMWS-  number  of  aircraft  (number  of  type  aircraft  in  the  squadron); 

RPF  -  ratable  pooling  factor  (number  of  part  failures  per  100  flying  hours  that  are 
repaired  at  the  location);  and 


10 


IMARPT  ~  intermediate  maintenance  activity  repair  time  (number  of  days 

between  the  time  of  failure  and  the  time  ready-for-issue  part  is  installed). 

(Cardillo,  personal  communication,  December  12,  2016) 

The  “90”  in  the  denominator  of  Equations  (1)  and  (2)  is  a  scaling  factor  to  convert  days 
to  quarters.  Equations  (1)  and  (2)  are  used  to  calculate  the  average  number  of  parts  that 
are  within  both  pipelines.  In  addition,  RBS  will  find  Total  Pipeline  by  summing  Resupply 
and  Repair  pipelines  and  this  value  will  be  used  to  calculate  the  EBOs  shown  in  Equation 
(3).  (Sax,  2012,  p.  30) 

b.  Expected  Backorders  Calculation 

Palm’s  Theorem  is  the  foundation  for  inventory  theory  of  repairable  NIINs. 
Sherbrooke  (2004,  p.  22)  states  its  “...importance  is  that  it  enables  us  to  estimate  the 
steady-state  probability  distribution  of  the  number  of  units  in  repair  from  the  probability 
distribution  of  the  demand  process  and  the  mean  of  the  repair  time  distribution.”  This 
implies  that  knowing  just  the  mean  of  the  repair  time  distribution,  and  not  the  distribution 
itself,  suffices.  EBO  is  calculated  as  a  function  of  the  inventory  positions  s  as  follows: 


oo 

E[BO,s]  =  2  (x-s) 

x=s+ 1 


e-pjpeun^  x  pipeiine< 
x! 


(3) 


The  x  in  the  Equation  represents  the  number  of  failures,  whereas  the  s  is  the  inventory 
position.  Pipeline  is  the  total  pipeline  (described  above).  E[BO,s]  calculates  expected 
backorders  by  NUN  for  candidate  files  (i.e.,  Access  database  that  contains  data  for 
multiple  factors  across  many  platforms  and  site  locations)  developed  by  the  NAVSUP 
WSS  Office  Code  N421  analyst  for  each  particular  site  or  platform.  Naturally,  as  5 
increases  E[BO;s]  decreases. 


c.  Supply  Delay  Calculation 

Once  E[BO;  s]  is  calculated,  the  next  step  for  the  RBS  approach  is  to  calculate  the 
average  amount  of  time  that  the  system  is  down  (i.e.,  supply  delay)  with  respect  to 
backorders  as  follows: 
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E[BO;  s] 


Supply  Delay  = 


WFHRS 

(MRF+RPF)xQPA  x  NUMWSx - 

2160 


(4) 


The  denominator  of  the  Supply  Delay  Equation  (4)  is  a  quarterly  unit  of  measure 
and  is  also  essential  in  calculating  the  system  operational  availability  seen  in  Equation  (5) 
(Cardillo,  personal  communication,  December  12,  2016).  The  2,160  in  Equation  (4)  is  the 
number  of  hours  per  quarter. 

d.  Item  Operational  Availability  Calculation 

The  calculation  in  Equation  (5)  is  a  key  component  for  the  RBS  approach  and 
is  necessary  to  determine  whether  a  system  is  operational  based  on  maintenance  and 
supply  requirements  (Sherbrooke,  2004,  p.  38).  NAVSUP  defines  Ao  for  a  given  system 
as: 


where: 


f 


\QPA 


Ao 


_ 1 _ 

i  (RemovalsxMTTR)+E[BO;s] 
+  NUMWSxQPA 


Removals  =  (  MPR+RPF  )  x  WFHRS  for  the  item; 


(5) 


NUMWS  =  number  of  type  aircraft  in  the  squadron;  and 

MTTR  =  mean  time  to  repair  the  WS.  (Sax,  2012,  p.  31) 

According  to  the  OPNAV  Instruction  444 1.5 A,  Ao  is  the  best  way  to  measure  readiness 
for  Navy  parts  associated  to  systems,  subsystems  and  equipment  essential  to  all  ship  and 
aircraft  missions  (Chief  of  Naval  Operations,  2011,  p.  3). 
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e.  Cost  to  Reduce  Supply  Delay  and  Cost  Effectiveness  Ratio  Ranked 
Shopping  List  ” 

Equation  (6)  shows  a  critical  calculation  made  by  most  RBS  approaches.  The 
equation  is  used  to  build  a  “Shopping  List”  by  ranking  each  NIIN’s  stock  level  (Cardillo, 
personal  communication,  December  12,  2016): 


Cost  Effectiveness  Ratio  =  _ Unit  Fnce _ (6) 

Decrease  In  Supply  Delayx  ( MRF  +  RPF )  x 

The  heuristic  rule  for  the  RBS-based  AVCAL  inventory  levels  calculates  the  cost 
effectiveness  ratio  for  different  values  of  s  for  all  items,  and  sorts  the  ratios  in  descending 
order.  The  shopping  list  begins  with  the  items  and  stock  levels  at  the  top  of  the  list,  until 
enough  items  have  been  added  to  reach  the  desired  Ao  (J.  Salmeron,  personal 
communication,  May  02,  2017). 

2.  NAVARM 

For  the  purpose  of  this  research,  NAVARM  will  be  considered  a  “black  box.” 
Furthermore,  this  research  is  only  interested  in  the  data  inputted  in,  and  the  direct  output 
from,  NAVARM.  NAVARM  was  developed  by  a  team  located  at  the  Naval  Postgraduate 
School  in  2016  in  response  to  a  NAVSUP  WSS  request  for  an  RBS  model  that  is  flexible 
and  transparent  in  its  methodology.  NAVARM  is  adjustable  by  means  of  dashboard 
settings  for  tolerance,  iterations,  and  maximum  solution  time.  The  NAVARM  RBS 
approach  applies  Equations  (1)  through  (6)  with  some  refinements  that  we  do  not  detail  in 
this  document.  NAVARM  uses  a  heuristic  optimization  to  calculate  NUN  allowances  that 
minimizes  total  cost  and  ensures  the  target  Ao  for  each  WS  is  satisfied.  NAVARM 
applies  to  single-site  and  multi-indentured  problems  (Salmeron,  2016). 

C.  ARROWS  SENSITIVITY  ANALYSIS 

In  2012,  Sax  conducted  an  SA  of  the  ARROWS  RBS  model  in  the  NAVSUP 
WSS  Readiness  Suite.  His  SA  is  different  from  the  one  developed  in  this  thesis,  but  it  is 
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significant  to  consider  while  performing  SA  on  NAY  ARM.  His  analysis  was  conducted 
on  both  RBS  and  RIMAIR,  and  the  inputs  were  adjusted  from  +10%  to  ±30%.  The 
inputs  that  were  part  of  the  ARROWS  SA  are  as  follows:  maintenance  rate  to  failure 
(MRF),  rotable  pool  factor  (RPF),  I-level  Turn-around  Time  (ITAT),  maintenance  cycles 
(OPTEMPO),  FMC,  wholesale  delay  time  (WDT),  and  Beyond  Capability  of 
Maintenance  (BCM,  described  below)  (Sax,  2012,  pp.  Appendix  1-1-2).  His  analysis 
consisted  of  two  SHORCALs,  one  amphibious  class  ship  and  one  aircraft  carrier.  This 
research  will  only  analyze  SA  associated  with  the  Aircraft  Carrier  (CV)  AVCAL. 

The  MRF  indicates  when  a  NUN  becomes  BCM  (i.e.,  failure  rate  for  parts  unable 
to  be  repaired  at  the  Organizational  (O)  or  Intermediate  (I)  Maintenance  Levels),  while 
RPF  is  the  rate  at  which  an  operating  site  can  repair  an  I-level  failure  (Sax,  2012,  p.  14). 
The  ITAT  is  the  number  of  days  it  takes  an  O  or  I-level  repairable  NUN  to  return  to  the 
organization’s  supply  system.  WDT  is  a  measure  of  days  from  the  time  of  requisition 
until  the  NUN  is  shipped  (Sax,  2012,  p.  24).  Noteworthy  in  this  analysis,  the  BCM  is  not 
an  ARROWS  model  input,  but  it  is  used  to  measure  the  overall  change  in  output  as  both 
MRF  and  RPF  are  adjusted.  (Sax,  2012,  p.  Appendix  1-1)  Next,  the  Operational  Tempo 
(OPTEMPO)  is  the  number  of  wartime  flying  hours  for  each  NUN  of  a  particular  WS 
(Sax,  2012,  p.  23).  Lastly,  the  FMC  factor  used  in  the  analysis  is  known  as  the 
Operational  Availability  (Ao)  (Sax,  2012,  p.  24.)  Each  WS  has  its  own  target  Ao,  and  as 
these  goals  are  varied,  the  output  is  recorded  and  presented  in  Figure  3. 
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Protection  and  endurance  are  not  pertinent  to  this  research  and  are  the  factors  used  for 
RIMAIR.  The  cell  for  protection  at  a  30%  increase  is  blank.  It  is  unclear  whether  or  not 
this  was  an  infeasible  setting  because  it  is  not  discussed  in  the  document,  nor  labeled  in 
the  image  used. 

Figure  3.  Results  from  SA  of  CV  AVCAL.  Source:  Sax  (2012). 


Figure  3  indicates  that  the  dominant  factors  are,  in  order  of  importance: 
OPTEMPO,  Rates  (i.e.,  combination  of  MRF  and  RPF),  andAo.  The  “dominant  factors” 
are  those  inputs  that  AVCAL  cost  is  influenced  by.  Sax  mentions  that  WDT  is  the  largest 
driver,  but  this  is  not  seen  in  Figure  3  (Sax,  2012,  p.  Appendix  1-3).  The  discrepancy  may 
be  explained  because  he  changed  days  by  percent  increments,  whereas  a  better  approach 
would  be  to  adjust  WDT  along  with  HPOST  by  a  sequential  integer  value.  As  WDT  is 
reduced  by  one  day,  it  can  reduce  the  value  of  an  AVCAL  by  3%,  which  is  very 
significant.  Sax  also  mentions  that  high  priority  order  and  ship  time  reacts  similarly  to 
WDT  because  both  measure  the  amount  of  time  in  days  it  takes  to  get  parts  into  the  hands 
of  customers  (Sax,  2012,  p.  Appendix  1-4). 

Some  aspects  taken  from  Sax’s  SA  on  CV  AVCAL.  The  factors  MRF,  RPF,  and 

have  a  nonlinear  relationship  with  the  cost  output,  whereas  the  rest  of  the  factors 

15 


appear  to  be  linear.  Sax  mentions  that  there  is  a  relationship  between  the  MRF  and  RPF 
given  they  are  both  used  to  calculate  the  pipeline  (Sax,  2012,  p.  Appendix  1-1).  However, 
it  is  not  obvious  how  those  factors  interact  with  each  other.  In  summary,  the  SA  study 
conducted  by  Sax  appears  to  use  one-factor-at-a-time  variation,  and  clearly  suggests  CV 
AVCAL  cost  factor  dominance. 
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III.  DATA  REVIEW  AND  METHODOLOGY 


If  you  can’t  fly  then  run,  if  you  can’t  run  then  walk,  if  you  can’t  walk  then 
crawl,  but  whatever  you  do,  you  have  to  keep  moving  forward. 

— Martin  Luther  King  Jr., 
civil  rights  activist 

In  Chapter  II,  we  explored  the  history  of  the  RBS  concept  and  its  importance  to 
the  U.S.  Navy.  This  chapter  will  discuss  data  review,  DOE,  and  SA.  These  are  three 
essential  steps  to  better  identify  NAVARM’s  most  influential  factors  on  cost.  This 
research  develops  NED,  a  tool  that  can  be  used  by  the  NAVSUP  WSS  analysis  team  to 
estimate  impacts  on  project  cost  given  factor  variability.  (See  Figure  4.) 
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Figure  4.  Research  design  flowchart 


17 


Figure  4  lays  out  the  four  steps  of  the  methodology,  starting  in  the  upper  left-hand 
comer.  First,  observe  the  blue  block  labeled  Candidate  File  Factors.  The  input  data  is 
collected  from  various  aviation  sites  from  both  Navy  and  Marine  Corps  aviation 
platforms.  The  key  factors  are  scaled  (orange  dashed  box)  by  multiplying  them  with  a 
portion  value  generated  using  the  Nearly  Orthogonal  and  Nearly  Balanced  (NOB)  mixed 
design  spreadsheet  NOB_Mixed_512DP_Vl.xlsx.  (Vieira,  2012)  Once  factors  are 
modified  the  Microsoft  Access  database  (used  for  the  baseline  scenario  provided  by 
NAVSUP)  is  renamed  and  saved,  therefore  maintaining  the  overall  integrity  of  the 
original  data  file. 

Second,  NAVARM  (black  box)  retrieves  the  newly  named  data  file  and  initiates 
its  RBS  solving  process. 

Third,  once  NAVARM  calculates  allowances  for  all  NIINs  and  cost,  RBS  cost  is 
extracted  from  the  NAVARM  RBS  worksheet  (light  blue  block)  and  saved  to  the 
spreadsheet  containing  the  NOB  factor  portions  (yellow  block).  This  step  matches  input 
and  output  data  (green  block). 

Fourth,  we  conduct  the  statistical  analysis  to  determine  the  impact  of  the  factors, 
as  well  as  fitting  a  regression  line  to  the  data  to  create  a  meta-model  that  estimates 
measured  output.  Finally,  NED  (red  box  in  bottom  right  of  Figure  4)  is  developed  for 
NAVSUP  WSS  Office  Code  N421  in  an  Excel  format  so  that  the  N421  analyst  team  can 
adjust  factors  and  see  how  they  influence  RBS  cost  for  each  site  location.  In  following 
the  methodology,  we  conducted  a  data  review  so  that  the  correct  DOE  is  applied. 

A.  DATA  REVIEW 

Before  developing  a  DOE,  this  research  investigated  multiple  candidate  files  (i.e., 
data  files  used  by  NAVSUP  WSS)  and  the  factors  that  we,  along  with  NAVSUP  WSS, 
consider  likely  to  be  significant.  The  data  review  provides  a  better  way  of  understanding 
the  factors  available  to  the  research  prior  to  conducting  DOE,  and  affords  us  with  the 
opportunity  to  identify  the  best  method  for  manipulating  data  fields  in  the  test  candidate 
files. 
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1. 


Factors  and  Various  Candidate  Files 


The  candidate  files  are  developed  by  NAVSUP  WSS  analyst  Office  Code  N421 
in  a  Microsoft  Access  database,  and  those  used  in  this  research  appear  in  Table  1. 


Table  1.  Database  candidate  files  by  location 


Test  Candidate  Name 

Candidate  File 

Description /Location 

HST 

A032420 1 6b- AVCAL  -HARRYSTRUMAN-20 160420144017 ,mdb 

USS  Hairy  S.  Truman  (CVN  75)  /  Norfolk,  VA 

BON 

A0623201 1  c-AVCAL-B  ONHOMMERTCH  .mdb 

USS  BonHomme  Richard  (LHD  6)  /  Sasebo.  Japan 

LEM 

A10252012d-REGIONAL-LEMOORE.mdb 

Naval  Air  Station  Lemoore  /  Lemoor,  CA 

BAT 

A1 1212012-AVCAL-BATAAN.mdh 

USS  Bataan  (LHD  5) /Norfolk,  VA 

NOR 

A042 120 1 6b-SASS-NORTHISLAN-20 1 6042 1 145227.mdb 

Naval  Air  Station  North  Island  /  North  Island,  CA 

MIS 

A06 1 920 14a-SH0RCAL-MISAWA.mdb 

Naval  Air  Facility  Misawa  /  Misawa,  Japan 

MAL 

A0706201  lb-CCSP-MALSl  l.mdh 

Marine  Aviation  Logistics  Squadron  1 1  /  San  Diego,  CA 

OCA 

A 1 0 1 420 14-SH0RCAL-0CEANA.mdh 

Naval  Air  Station  Oceana  /  Virginia  Beach,  VA 

DEN 

A042020 1 6-FMS-DENMARK-Conf-20 1 60420 1 23S00 1  .mdb 

Danish  Naval  Ah  Squadron  /  Denmark 

IWO 

A01 1 12017-AVCAL-IWOJIMA-NAVARM.mdb 

USS  Iwo  Jima  (LHD  7)  /  Mayport,  FL 

The  candidate  files  will  be  referred  to  by  their  test  candidate  name  when 
discussed  in  both  chapters  III  and  IV.  Table  1  describes  the  platform  and  location  for 
each  candidate  file  by  description  and  location  category.  We  have  a  wide  range  of 
platforms  from  shore  to  sea,  as  well  as  aviation  data  that  spans  from  west  to  east  coast. 

To  begin,  the  factor  discussion  will  use  the  USS  Harry  S.  Truman  (HST)  test 
candidate  name  to  show  its  key  tables  along  with  each  factor’s  definition.  Figure  5 
displays  the  tables  ArrowsCandidate,  ArrowsParamSW,  and  ArrowsParamWS,  which 
contain  all  of  the  factors  we  use  in  this  research.  We  omit  additional  figures  of 
ArrowsParamSW  and  ArrowsParamWS  tables,  but  will  list  those  factors  that  can  be 
found  in  each. 
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Figure  5.  HST  test  candidate  file  identifying  required  tables 

The  factors  located  in  ArrowsParamSW  table  are  NUMWS,  Ao,  and  WFHRS. 
ArrowsParamWS  contains  the  MTTR  factor  only.  Figure  6  displays  the  ArrowsCandidate 
table,  which  contains  the  following  factors:  QPA,  IMA_RPR_TM  (also  known  as 
IMARPT),  LP_OST  (also  known  as  LPOST),  HP_OST  (also  known  as  HPOST), 
WHSL_DELAY  (also  known  as  WDT),  UNITPRICE,  MRF,  and  RPF.  In  addition,  it 
contains  two  factors  not  seen  in  Figure  6:  EXP_PRG_W  (also  known  as  EXPWFHRS), 
and  FLY_HRS  (also  known  as  FHRS).  Note:  NAVARM  also  uses  the  ArrowsParamCS 
table  in  its  calculations,  but  that  table  does  not  contain  any  factors  for  this  research. 
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Figure  6.  HST  test  candidate  file  factors  in  ArrowsCandidate  table 


For  reporting  purposes,  we  also  show  the  number  of  NIINs  in  each  candidate  file. 
The  number  of  NIINs  and  number  of  WS  will  vary  per  candidate  file.  (See  Table  2.) 
Neither  one  is  a  factor  in  our  DOE.  They  are  fixed  parameters  associated  with  each  case. 
The  number  of  NIINs  shown  in  Table  2  includes  RBS-only  items. 
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Table  2.  Baseline  candidate  file  specifications 


Test  Candidate  Name 

#  of  RBS  NIINs 

#  of  WS  Type 

A0  Target 
Range*  (%) 

HST 

11,204 

7 

59-65 

BON 

4,145 

7 

65-82 

LEM 

77,209 

23 

46-58 

BAT 

5,777 

7 

65-80 

NOR 

501 

1 

63 

MIS 

2,374 

3 

53-66 

MAL 

30,181 

7 

59-75 

OCA 

35,586 

10 

46-58 

DEN 

3,379 

1 

85 

IWO 

2,683 

6 

65-80 

*Note:  Ao  range  is  for  cases  with  multiple  WS. 


a.  Factor  Definitions 

The  next  step  in  completing  the  data  review  is  to  briefly  define  each  factor  used  to 
identify  NAV ARM’s  output  sensitivity.  All  factors  defined  below  will  have  their  baseline 
values  adjusted  within  a  range  of  ±10% . 

•  The  factor  EXP_PRG_W  [expanded  war  flying  hours]  is  the  quarterly 
wartime  flying  hours  for  a  particular  item  within  a  certain  WS.  The 
expanded  war  flying  hours  are  determined  by  dividing  a  given 
maintenance  cycle  rate  by  100  for  each  NUN  in  a  WS.  This  value  indicates 
the  overall  population  of  the  NUN  for  that  WS.  (Oswald  et  al.,  2015,  p.  6) 

•  The  factor  QPA  [Quantity  Per  Application]  is  the  total  quantity  of  each 
NUN  for  each  WS.  (Oswald  et  al.,  2015,  p.  6) 

•  The  factor  IMA_RPT_TM  [intermediate  maintenance  activity  repair  time] 
represents  the  days  necessary  to  receive  a  NUN  from  organizational 
maintenance  plus  the  time  required  for  scheduling  and  repairing  the  part  at 
the  intermediate  maintenance  facility.  This  assumes  that  the  essential  part 
to  be  repaired  is  available  in  the  system.  (Oswald  et  al.,  2015,  p.  7) 
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Both  factors  LPjOST  [Low  Priority  Order  and  Ship  Time]  and  HP_OST 
[high  priority  order  and  ship  time]  are  the  number  of  days  required  to  ship 
a  low-  and  high-priority  NUN,  respectively,  from  the  supply  system  during 
the  requisitioning  process  (Oswald  et  al.,  2015,  p.  8).  Both  factors  are 
highly  correlated;  therefore,  the  low  priority  factor  is  dropped  from  this 
research.  Although  the  high  priority  factor  appears  discrete,  for  the 
purpose  of  this  study,  we  vary  it  by  percentage  like  all  the  other  factors. 

The  factor  WHSL_DELAY  [wholesale  delay  time ]  represents  the  number 
of  days  required  for  the  wholesale  system  to  make  a  ready -for-issue  part 
available  to  satisfy  a  demand  at  the  customer  level.  (Oswald  et  al.,  2015,  p. 
8) 

The  factor  UNITPRICE  [Unit  Price]  represents  the  price  for  each  NUN. 
(Oswald  et  al.,  2015,  p.  10) 

The  factor  MRF  [maintenance  rate  to  failure]  represents  the  number  of 
failures  for  each  NUN  that  cannot  be  repaired  at  the  site  location  “per 
flying  hour  (or  maintenance  cycle)  per  item  installed.”  (Oswald  et  al., 
2015,  p.  10) 

The  factor  RPF  [rotable  pooling  factor]  denotes  the  number  of  part 
failures  that  are  repaired  at  each  site  location  per  flying  hour.  (Oswald  et 
al.,  2015,  p.  10) 

The  factor  FLY_HRS  [flying  hours]  represents  the  length  of  use  for  each 
part  and  it  can  be  used  to  determine  a  part’s  rate  of  failure. 

The  factor  MTTR  [mean  time  to  repair]  identifies  the  organization’s 
maintenance  hours  required  to  restore  a  failed  WS  back  to  operating. 
(Oswald  et  al.,  2015,  p.  12) 

The  factor  WS_number  [number  of  aircraft]  specifies  the  number  of 
aircraft  to  support  a  specific  WS.  (Oswald  et  al.,  2015,  p.  13) 
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•  The  factor  RBS_RDGoal  [ RBS  performance  goal]  is  also  known  as  the 

goal,  which  is  a  percentage  used  to  represent  the  targeted  FMC. 
(Oswald  et  al.,  2015,  p.  14) 

•  The  factor  WAR_FHRS  [wartime  flying  hours]  is  the  number  of  “aircraft 
times  the  flying  hours  per  quarter  per  aircraft”  in  a  wartime  scenario. 
(Oswald  et  al.,  2015,  p.  15) 

b.  Factor  Correlations 

We  construct  a  correlation  matrix  in  the  statistical  software  JMP  (2017)  to 
identify  whether  there  are  any  highly  correlated  factors  other  than  the  previously 
mentioned  LP_OST  and  HP_OST.  Observing  Figure  7  reveals  multiple  factors  that  have 
a  strong  positive  or  negative  correlation.  For  example,  the  factors  QPA  and 
EXP_PRG_W  have  a  correlation  of  0.96,  RBS_RDGOAL  and  MTTR  have  a  correlation 
of  0.81,  and  RBS_RDGOAL  and  HP_OST  have  a  correlation  of  -0.99. 
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Figure  7.  Factor  correlation  matrix  for  the  HST  candidate  file 

The  DOE  developed  for  this  research  seeks  to  determine  the  interaction  between 
factors  in  order  to  estimate  the  NAVARM  output  (specifically  cost)  as  a  function  of 
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changes  in  the  factors.  Specifically,  we  will  estimate  AVCAL  RBS  cost  with  factors 
ranging  between  ±10%  of  their  base  value  (i.e.,  from  their  nominal  value  in  the  specific 
candidate  file  provided  by  NAVSUP).  Section  C  of  this  chapter  provides  a  more  in-depth 
discussion  of  the  SA  techniques  in  regards  to  the  NOB  DOE. 

2.  NAVARM  Output 

The  last  pieces  of  the  data  to  be  reviewed  in  this  research  are  the  required 
dependent  variables.  As  each  factor  (independent  variable)  defined  previously  is 
modified,  the  RBS  cost  and  time  for  a  NAVARM  RBS  solution  will  be  collected.  Figure 
8,  features  two  sections:  the  left  side  is  the  NAVARM  Dashboard,  and  the  right  side  of 
the  figure  is  the  RBS  solution  worksheet.  NED  focuses  on  RBS  best  cost  (incased  in  the 
green  enclosed  box  on  the  left  side).  We  collect  the  total  time  to  obtain  the  solution 
(incased  in  the  yellow  enclosed  box  on  the  left  side)  located  within  the  dashboard  as  well, 
but  we  use  it  for  internal  purposes  to  track  progress  of  the  DOE  trial  runs.  In  the  DOE, 
the  dependent  variables  are  matched  to  corresponding  independent  variable  changes  for 
its  specific  trial.  A  complete  explanation  of  how  this  is  accomplished  is  discussed  next. 
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Figure  8.  NAVARM  split  screen  of  output  data  collected 


B.  EXPERIMENTAL  DESIGN 

Before  explaining  the  DOE,  the  standard  settings  of  NAVARM  will  be  discussed 
for  each  trial.  The  following  discussion  identifies  the  most  effective  settings  in 
NAVARM  in  preparation  for  the  DOE  simulation  trial  runs.  Parameter  settings  in 
NAVARM  will  remain  the  same  for  all  trial  runs  to  maintain  consistency  in  the 
experimental  design. 


1.  NAVARM  Configuration 

Standard  settings  for  NAVARM  trial  runs  appear  in  Figure  9,  except  as  noted 
below.  A  mix  of  settings  is  available.  Some  are  not  related  to  performance.  Others  are 
intended  to  strike  a  balance  between  time  spent  and  solution  quality  (J.  Salmeron, 
personal  communication,  May  02,  2017). 
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Figure  9.  NAY  ARM  Dashboard  DOE  simulation  standard  configuration 


The  focal  areas  to  setup  NAVARM  for  this  research  appear  in  the  yellow  and  blue 
boxes  of  Figure  9.  To  start,  the  yellow  boxes  are  the  file  names  of  the  candidate  file 
containing  the  required  factors  for  that  specific  experiment.  In  this  case,  they  are  named 
NED,  because  the  original  candidate  file  must  remain  unchanged  for  future  experimental 
trials.  Outlined  with  yellow  boxes,  the  candidate  file  name  and  its  site  identification 
(SITE_ID)  are  entered  in  columns  H  and  M  under  the  Database  filename  section  of 
NAVARM  Dashboard.  Again,  we  enter  the  file  name  and  SITE_ID  under  column  C  in 
their  second  location  next  to  their  Case  #  that  the  user  inputs.  In  this  case,  the  file  and 
SITE_ID  are  case  number  17. 

The  three  settings  (shown  in  the  blue  box)  controlling  the  length  in  a  NAVARM 
run  are  “ Maximum  Time  (minutes),”  “ Maximum  #  of  Main  Passes,”  and  “Max.  #  of 
Refinement  &  Polish  per  Pass.”  The  Maximum  Time  (minutes)  is  the  time  limit  allowed 
for  NAVARM  to  find  a  solution,  including  the  time  other  tasks  (such  as  data  preparation. 


27 


RIMAIR  execution,  etc.).  The  special  case  of  zero  is  used  to  set  an  unlimited  amount  of 
time,  and  this  is  the  default  choice  for  NED.  The  Maximum  #  of  Main  Passes  relates  to 
the  number  of  global  iterations  for  NAVARM  to  find  a  solution  for  the  RBS  portion  of 
the  model.  More  passes  may  produce  a  better  solution,  but  will  require  more  time.  The 
default  value  for  this  input  is  ten,  but  we  changed  it  to  five  for  NED  in  order  to  reduce 
run  time.  The  Max.  #  of  Refinement  &  Polish  per  Pass  is  used  to  refine  the  solution,  and 
the  input  value  for  this  setting  is  ten.  Again,  the  larger  this  value  is,  the  longer  it  will  take 
NAVARM  to  solve  RBS. 

Note:  Run  time  may  vary  by  computer.  The  processor  used  in  this  research  is  an 
Intel  (R)  Atom  (TM)  x7-Z8700  with  a  1.6  GHz  CPU,  and  it  takes  approximately  seven  to 
fifteen  minutes  for  NAVARM  to  produce  a  solution,  depending  on  the  candidate  file. 

2.  Simulation  by  Visual  Basic  for  Applications 

Considering  that  NAVARM  is  a  tool  developed  and  operated  in  Microsoft  Excel 
and  Visual  Basic  for  Applications  (VBA),  we  develop  a  set  of  VBA  subroutines  that 
conduct  a  simulation  with  the  NAVARM  model.  The  following  is  a  list  of  steps  taken  to 
conduct  the  NAVARM  simulation  based  on  the  NOB  input  values: 

•  The  first  step  is  to  select  suitably  scaled  values  from  the  NOB  spreadsheet, 
and  record  those  values  in  a  workbook  named  NED.xlsm.  The  use  of  the 
latter  spreadsheet  will  be  discussed  more  in  Section  C,  subparagraph  2  of 
this  chapter. 

•  Second,  a  subroutine  named  fileNED  in  the  spreadsheet  NED.xlsm  will 
access  the  specified  candidate  file  and  change  property  Field  Size  in 
Microsoft  Access  to  a  “double”  (i.e.,  floating-point  that  handles  most 
decimal  numbers)  so  that  each  data  field  can  be  manipulated. 

•  Third,  the  subroutine,  named  LHSscalar,  retrieves  the  scaled  values  for  all 
thirteen  factors  defined  above.  Structured  Query  Language  (SQL)  is  used 
to  open  the  Access  database  and  modify  each  field  for  each  factor  with  the 
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NOB  value  in  the  workbook  named  NED.xlsm.  SQL  then  closes  the 
database  and  saves  changes  made  to  the  factors. 

•  Fourth,  the  last  subroutine  named  runRBS  would  open  NAVARM,  input 
the  candidate  file  name  along  with  SITE_ID,  and  then  launch  NAVARM. 
Once  NAVARM  establishes  a  solution  for  that  trial  the  subroutine  copies 
the  best  cost  value  and  the  time  it  takes  NAVARM  to  solve  (for  internal 
use  only  to  track  the  simulation). 

•  Finally,  we  wrap  the  subroutines  with  a  for  loop  that  iterates  through  all  of 
the  design  points  that  are  defined  by  the  NOB  DOE.  Once  the  for  loop 
reaches  the  end  of  the  NOB  design,  we  conduct  regression  analysis  on  the 
data  created  with  new  inputs  and  measurable  output  (NAVARM  RBS 
cost).  Also,  with  an  understanding  of  the  data  and  process  of  simulation, 
the  research  helps  determine  the  best  method  of  measuring  factor 
dominance  as  well  as  regression  analysis  with  the  newly  developed  data. 

C.  SENSITIVITY  ANALYSIS  TECHNIQUES 

SA  is  a  method  for  assisting  the  decision  makers  in  determining  future  differences 
while  continuing  to  shape  their  current  policies  or  business  rules.  SA  requires  data  that 
provides  us  with  the  ability  to  investigate  the  designated  dependent  and  independent 
variables.  We  conducted  SA  upon  completing  multiple  DOEs  discussed  later  in  this 
section  using  a  NOB  design  that  captures  changes  in  AVCAL  costs  as  independent 
variables  vary. 

We  used  the  following  SA  techniques:  One-Factor-at-a-Time  (OAT)  analysis, 
scatter  plots  analysis,  and  regression  analysis.  Stepwise  regression  facilitates  construction 
of  predictive  meta-models,  which  are  the  basis  of  NED. 
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1. 


One-F  actor-at-a-Time 


The  OAT  is  a  historical  method  used  to  identify  main  effects.  It  adjusts  one  factor 
at  a  time  while  keeping  all  other  factors  constant.  We  use  OAT  in  this  thesis  to  provide  a 
basis  for  comparison  with  prior  work. 

The  OAT  design  is  based  on  Equation  (7).  The  length  is  determined  by  the  number 
of  variations  made  to  each  factor  and  the  number  of  factors.  Each  factor  is  varied  up  to 
±99%  in  increments  of  10%.  The  final  increment  is  9%  to  avoid  errors  generated  if  the 
factors  are  zero  or  too  large.  The  k  in  Equation  (7)  is  the  number  of  factors  to  be  examined. 
(Saltelli  et  al.,  2000,  p.  68)  The  value  20  is  the  number  of  levels  for  each  factor. 


OATdesign  =  20  k  +  1  ^  ^ 

As  a  result,  the  overall  OAT  design  will  consist  of  261  trials  based  on  13  factors. 

After  completing  the  OAT  trials,  SensitivityRank,  defined  in  Equation  (8),  will 
determine  factor  ranking: 


SensitivityRank  = 


|  Pciya -  Para n 
Para . „ 


(8) 


where 


Parama  =  Maximum  value  of  the  measured  output  (RBS  cost) 

Paramin  =  Minimum  value  of  the  measured  output  (RBS  cost) 

SensitivityRank  yields  a  number  between  zero  and  one  (Saltelli  et  ah,  2000,  p. 
176).  A  value  closer  to  one  indicates  high  output  variation,  while  a  value  closer  to  zero 
indicates  the  minimal  influence  on  the  output.  This  analysis  reflects  the  interest  in 
NAVARM  RBS  cost. 
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2.  Design  of  Experiments 

SA  alone  cannot  identify  the  most  influential  factors,  but  a  well-crafted  DOE  can. 
Saltelli,  et  al.  (2000)  state: 

Although  there  are  several  differences  between  physical  and  simulation 
experiments,  sensitivity  analysis  is  based  on  the  same  principles  as  those 
underlying  DOE.  The  selection  of  inputs  at  which  to  run  a  computer  code 
is  still  an  experimental  design  problem,  and  statistical  ideas  for  design  are 
helpful  (Sacks  et  al.,  1989a).  Further,  much  of  the  terminology  used  in  SA 
has  originated  in  a  DOE  setting,  (p.  51) 

Sanchez  and  Wan  (2015,  p.  1798)  discuss  why  OAT  may  be  ineffective,  since  it  ignores 
the  potential  for  factor  interactions.  A  well-designed  experiment  explores  combinations 
of  factors  that  can  reveal  possible  relationships  that  OAT  ignores. 

a.  Benefits  of  using  Space-filling  Nearly  Orthogonal  and  Nearly  Balanced 

We  used  NOB  design  to  vary  the  factors.  The  NOB  methodology  is  applicable  for 
the  following  reasons: 

•  Latin  Hypercube  sampling  is  highly  flexible  and  allows  the  experimenter 
to  span  the  factor  space  with  a  sample  size  that  compares  favorably  to  that 
of  a  fractional  factorial  design.  (Sanchez  and  Wan,  2015,  p.  1803) 

•  According  to  Vieira,  the  NOB  is  a  mixed  design  that  is  balanced  and 
orthogonal  for  all  factor  types  and  levels.  It  has  “low  maximum  absolute 
pairwise  correlation  and  imbalance.”  (Vieira  et  al.,  2013,  p.  273) 

•  NOB  sampling  has  “good  space-filling  and  orthogonality  behavior.” 
(Vieira  et  al.,  2011,  p.  3608) 

Latin  hypercubes  provide  good  estimation  of  factor  effects  with  low  variance 
(Saltelli  et  al.,  2000,  p.  22).  Appendix  A  contains  the  correlation  matrix  and  scatterplots 
for  the  NOB.  Note  that  there  is  nearly  zero  correlation  among  all  factors. 
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b.  Scatter  Plots 

Scatter  plots  are  often  used  to  try  to  visualize  the  relationship  between  the 
dependent  variable  and  the  factors,  but  the  reader  should  note  that  they  can  be  misleading 
in  high  dimensional  cases  where  projecting  to  lower  dimensions  can  mask  effects. 
Regression  is  far  more  reliable  (Saltelli  et  al.,  2008,  pp.  17-20).  As  an  example,  scatter 
plots  for  the  HST  candidate  file  are  presented  in  Chapter  IV  Section  B. 

c.  Regression 

The  NOB  affords  us  the  ability  to  assess  the  influence  of  each  factor  on 
performance  measures  using  regression  analysis.  Stepwise  regression,  a  well-known 
technique,  efficiently  allows  us  to  construct  meta-models.  Figure  10  shows  diagnostic 
information  that  can  be  used  to  assess  the  quality  of  the  model  fit  for  the  HST  test  case. 
After  determining  which  factors  are  most  influential  from  this  assessment,  the  final  step 
is  to  generate  the  prediction  formula  for  NAVARM. 

The  resulting  regression  model  is  presented  in  Figure  11.  In  this  case,  the 
Prediction  Expression  for  HST  shows  that  the  meta-model  has  only  main  effects  when 
estimating  the  NAVARM  RBS  cost.  The  coefficients  for  each  factor  are  all  positive 
except  the  factor  WS_number,  which  shows  a  negative  correlation  relationship  to  RBS 
cost.  We  apply  this  process  to  the  other  nine  test  candidate  files  using  the  statistical 
software  JMP  (2017). 
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w  Response  RBS  Total  Cost 
>  Effect  Summary 
A  Actual  by  Predicted  Plot 


60000000  80000000  100000000  120000000 
RBS  Total  Cost  Predicted  Pc.0001  RSq=0.99 
RMSE=1.4e+6 


A  Summary  of  Fit 

RSquare  0.986296 

RSquare  Adj  0.985442 

Root  Mean  Square  Error  1 375272 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 

A  Analysis  of  Variance 


a  Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

30 

6.5478e+16 

2.183e+1 5 

1153.976 

Error 

481 

9.0975e+14 

1.891e+12 

Prob  >  F 

C.  Total 

511 

6.6388e+16 

<.0001* 

Figure  10.  Stepwise  regression  results  example 


Prediction  Expression 

-566966909.9203 
+  2926935.78815754*  HP_OST 
+  32288847.0199892*  WHSL_DELAY 
+  181591141.901322*  UNITPRICE 
+  155763845.856915*  MRF 
+  152125728.399188*  WAR_FHRS 
+  -51297727.194505*  WS.number 
152646655.673763*  RBS_RDGOAL 

Figure  11.  Stepwise  regression  prediction  formula 
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IV.  ANALYSIS 


Statisticians,  like  artists,  have  the  bad  habit  of  falling  in  love  with  their 

models. 

— George  E.P.  Box, 

British  statistician 

In  Chapter  III,  we  discussed  the  methodology  for  developing  data  using  the  ten 
test  candidate  files.  This  Chapter  analyzes  how  NAVARM’s  RBS  cost  output  is  sensitive 
to  different  factor  variations.  The  regression  results  are  assessed  using  four  statistical 
measures:  R-Square  adjusted.  Root  Mean  Square  Error  ( RMSE ),  F  ratio,  and  t  Ratio. 
(Cleary  and  Levenbach,  1982,  pp.  43-51)  We  only  display  the  meta-model  results  for 
HST  test  candidate  file  in  Section  C,  subparagraph  1  of  this  chapter.  In  Appendix  D,  we 
provide  the  remaining  nine  test  candidate  file  meta-model  results. 

A.  ONE-FACTOR-AT-A-TIME  RESULTS 

We  experimented  with  the  OAT  design  for  a  few  of  the  test  candidate  files  prior 
to  conducting  the  NOB  DOE  to  see  if  any  factors  largely  affect  NAVARM  RBS  cost. 
This  method  is  intended  to  be  informative  in  observing  how  sensitive  NAVARM  RBS 
cost  is  to  each  factor.  We  conducted  OAT  design  in  five  of  the  ten  test  candidates’  files 
listed  in  Table  1:  HST,  MIS,  BON,  OCA  and  BAT.  The  OAT  experimentation  resulted  in 
a  similar  conclusion  among  all  site  locations.  This  result  only  changes  one  factor  at  a  time 
without  interactions.  The  sensitivity  results  (Figure  12)  display  HST  RBS  cost  as  a 
function  of  changes  to  the  baseline  values. 

It  is  worth  noting  that  the  cost  of  HST  allowances  appears  to  increase 
exponentially  as  the  RBS_RDGOAL  (baby  blue)  factor  increases.  However,  as  the  factor 
WS_number  (number  of  aircraft)  is  reduced  there  appears  to  be  a  negative  effect  on  RBS 
cost.  Additional  SA  graphs  of  the  four-other  site  locations  are  in  Appendix  B.  The  graphs 
capture  each  factor  change  as  it  is  increased  or  decreased  from  its  candidate  file  baseline 
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value.  However,  they  do  not  inform  us  which  factors  are  most  influential,  nor  do  they 
identify  interaction  effects. 


HST  OAT  Sensitivity  Analysis  of  RBS  Cost 


$300 
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§  $150 
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H  $50 
$0 


CV  Cy  Q-  <5r  CJ-  Q/  Q-  Q-  Cy  Qr  \-  v  \-  V  \-  \-  \-  V  \-  v  V 


Percent  Change  from  Baseline 


QPA  IMA_RPR_TM  EXP_PRG_W 

WHSL_DELAY^— UNITPRICE  MRF 

FLY_HRS  MTTR  WAR_FHRS 

RBS_RDGOAL 


HP_OST 

RPF 

S_number 


Figure  12.  HST  OAT  SA  for  RBS  cost 

B.  SCATTER  PLOT  RESULTS 

Like  the  OAT  design,  the  scatter  plots  are  often  used  to  assess  the  sensitivity  of 
NAVARM  RBS  cost  as  given  factors  change  in  value.  Two  scatter  plots,  along  with  fitted 
lines  for  the  factors  MRF  and  RFP,  are  shown  in  Figure  13.  The  rest  of  the  scatter  plots 
for  each  factor  can  be  seen  in  Appendix  C.  The  scatter  plots  are  a  visual  tool  to  show  how 
one  factor  reacts  to  the  output  and,  in  this  case,  to  RBS  cost.  The  formulae  created  for  the 
one  factor  Bivariate  Fit  in  Figure  13a  and  Figure  13b  are  not  useful  in  making  predictions 
for  the  entire  model,  but  they  can  provide  estimates  for  individual  factor  main  effects. 
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▼  Bivariate  Fit  of  RBS  Total  Cost  By  MRF 

130000000 

120000000 


MRF 


-Linear  Fit 


21  Linear  Fit 

RBS  Total  Cost  =  14776455  +  70633504*MRF 
21  Summary  of  Fit 
RSquare  0.128758 

RSquare  Adj  0.127049 

Root  Mean  Square  Error  10649476 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 

2)  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  8.5479e+15  8.548e+15  75.3709 

Error  510  5.784e+16  1.134e+14  Prob  >  F 

C.  Total  511  6.6388e+16  <.0001* 

2)  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  14776455  8149561  1.81  0.0704 

MRF  70633504  8135960  8.68  <.0001* 


^  ▼  Bivariate  Fit  of  RBS  Total  Cost  By  RPF 

130000000 


0.9  0.92  0.94  0.96  0.98  1  1.02  1.04  1.06  1.08  1.1 

RPF 


-Linear  Fit 


0.009705 
0.007764 
11353792 
85409959 
512 


21  Linear  Fit 

RBS  Total  Cost  =  66017653  +  19392306*RPF 

2!  Summary  of  Fit 

RSquare 
RSquare  Adj 
Root  Mean  Square  Error 
Mean  of  Response 
Observations  (or  Sum  Wgts) 

21  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  6.443 1e+ 14  6.443e+14  4.9982 

Error  510  6.5743e+16  1.289e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.0258* 

-  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  66017653  8688543  7.60  <.0001* 

RPF  19392306  8674042  2.24  0.0258* 


a.  MRF  by  RBS  cost 


b.  RPF  by  RBS  cost 


Figure  13.  HST  bivariate  fit  of  two  factors  by  RBS  cost 


C.  STEPWISE  REGRESSION  MODEL  RESULTS 

OAT  is  a  mediocre  design  for  identifying  the  factor  effects,  and  scatter  plots 
provide  minimal  insight  on  factor  effects.  Stepwise  regression  will  best  identify 
NAVARM  RBS  cost  sensitivity  and  provide  us  a  capability  in  building  our  best  fit  meta¬ 
models  that  will  make  predictions  for  all  factor  variations. 

1.  Meta-model  Fit 

Finding  the  meta-model,  using  the  stepwise  regression  process  discussed  in 
Chapter  III,  is  the  focus  of  this  section.  The  HST  test  candidate  file  data  is  used  to 
illustrate  the  meta-model  fitting  for  the  rest  of  the  experiment.  Also,  the  stepwise 
regression  results  for  NAVARM  RBS  cost  are  explained  in  detail  for  developing  a 
practically  significant  meta-model. 

Additionally,  the  nine  other  test  candidate  files  have  been  analyzed  using  the 


same  process  as  HST.  Their  statistical  summaries  are  available  in  this  Section  under 
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Subparagraph  2,  but  their  meta-models  can  be  seen  in  Appendix  D.  The  meta-models  are 
developed  by  starting  with  all  main,  two-way  interaction,  and  quadratic  effects.  With  13 
factors,  there  are  78  (13  choose  2)  two-way  interactions  plus  26  (13  times  2)  main  and 
quadratic  effects.  The  number  of  potential  terms  is  thus  104. 

The  stepwise  regression  will  assess  all  terms,  and  while  stepping  through  them, 
find  those  that  are  statistically  significant  for  the  data.  In  Figure  14,  the  stepwise  function 
for  the  HST  test  candidate  file  finds  only  22  effects  out  of  the  104  that  are  statistically 
significant  in  developing  the  model.  We  choose  those  with  a  t  Ratio  greater  than  ten 
because  we  deem  them  “practically  significant.”  Those  effects  on  the  lower  end 
(highlighted  in  red  box  in  Figure  14)  have  less  effect  on  the  outcome,  and  we  judged  that 
estimation  power  principally  lies  in  those  nine  main  effects. 


Sorted  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>|t| 

RBS.RDGOAL 

117207984 

1086834 

107.84 

pi 
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MRF 
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6.17 

l 

<.0001* 

(RBS_RDGOAL-1  )*(RBS_RDGOAL-1) 

93752538 

21140391 
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0.0005’ 
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3.46 
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3.30 
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3.17 
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3.12 

0  0019* 

(HP_OST-1)*(M7TR-1) 

-60147377 
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-3.06 

0,0024’ 
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54169763 

18594104 
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0.0065’ 
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Figure  14.  HST  stepwise  regression  results  for  main,  two-way  interactions,  and 

quadratic  effects. 


Figure  15  shows  that  the  reduction  in  effects  from  the  bounds  set  on  the  t  Ratio  is 
minimal.  Figure  15a  displays  the  meta-model  with  all  statistical  significant  effects 
selected  by  stepwise  regression.  Figure  15b  displays  a  meta-model  with  only  main  effects 
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(no  two-way  interactions  or  quadratic  effects)  that  have  a  t  Ratio  greater  than  ten,  as 
discussed  previously.  Starting  at  the  top,  observing  both  Figures  15a  and  15b,  the  Actual 
by  Predicted  Plot  shows  meta-models  that  have  a  tight  grouping  of  data  points  with  a 
prediction  line  (red)  that  passes  precisely  through  the  center  of  the  grouping  with  minimal 
variation  between  points,  hence  the  large  R-Squares  adjusted.  In  fact,  both  R-squares 
adjusted  are  nearly  the  same,  the  RMSE  are  only  slightly  different,  and  the  reduced  meta¬ 
model  in  Figure  15b  has  an  F  ratio  nearly  twice  that  of  the  full  meta-model  in  Figure  15a. 
This  reduction  in  the  number  terms  included  in  the  meta-model  does  not  noticeably 
reduce  the  effectiveness  of  the  meta-model  itself,  based  on  observed  plots  and  statistical 
summaries. 


Actual  by  Predicted  Plot 


60000000  80000000  100000000  120000000 
RBS  Total  Cost  Predicted  RMSE=1 .4e+6  RSq=0.99 
PValue<.0001 


Summary  of  Fit 

RSquare  0.985384 

RSquare  Adj  0.984727 
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Analysis  of  Variance 
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a.  HST  Meta-model  (Full) 


Actual  by  Predicted  Plot 


RBS  Total  Cost  Predicted  RMSE=1 .6e+6  RSq=0.98 
PValue<.0001 

Summary  of  Fit 

RSquare  0.979434 

RSquare  Adj  0.979065 

Root  Mean  Square  Error  1 649191 
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Observations  (or  Sum  Wgts)  512 

Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

9 

6.5022e+16 

7.225e+15 

2656.306 

Error 
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1.3654e+15 

2.72e+12 

Prob  >  F 

C  Total 
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b.  HST  Meta-model  (Reduced) 


Figure  15.  HST  meta-models  Actual  by  Predicted  with  statistical  summaries 


To  identify  outliers,  Studentized  Residual  plots  are  displayed  for  HST  NAVARM 
RBS  cost  in  Figure  16.  As  a  check  and  balance,  we  conducted  Studentized  Residual  plots 
for  all  test  candidate  file  meta-models  (full  and  reduced),  and  they  are  available  in 
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Appendix  D.  In  Figures  16a  and  16b,  the  data  points  appear  tightly  fit  on  the  centerline 
(blue  horizontal  line)  leading  us  to  determine  that  neither  meta-model  has  outliers. 


Studentized  Residuals 

—  A  J 


0  50  100  150  200  250  300  350  400  450  500 
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a.  HST  Meta-model  (Full) 


Studentized  Residuals 
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b.  HST  Meta-model  (Reduced) 


Figure  16.  HST  meta-models  studentized  residual  plots 


For  simplicity  and  practicality,  we  decided  to  use  the  reduced  meta-model  with 
the  main  effects  only.  The  remaining  nine  test  candidate  files  were  developed  using  the 
same  technique  described  for  the  HST  test  candidate  file.  While  developing  the  meta¬ 
models  it  is  notable  that  60%  of  the  models  developed  have  only  main  effects  (no  two- 
way  interactions  or  quadratic  effects).  However,  there  are  four  test  candidate  files  that 
have  a  quadratic  effect  (RBS_RDGOAL  x  RBS_RDGOAL).  The  test  candidate  files  that 
have  the  RBS_RDGOAL  quadratic  effect  are  BAT,  BON,  IWO,  and  DEN.  The 
interesting  characteristic  about  these  four  test  candidate  files  is  that  they  are  for  sites  with 
rotary  wing  aircraft  parts.  However,  we  cannot  conclude  that  rotary  wing  aircraft  cause 
this  effect. 

Exponential  and  reciprocal  transformations  of  the  factor  RBS_RBGOAL  show  no 
improvement  to  the  overall  meta-model  development  for  those  with  non-linearity.  In  fact, 
both  of  those  transformations  on  RBS_RDGOAL  cause  R- Square  adjusted  to  decrease, 
RMSE  to  increase,  F  ratio  to  decrease,  and  t  Ratio  to  decrease  compared  to  the  non- 
transformed  meta-models,  indicating  that  the  quadratic  fits  best  among  these  choices. 

Finally,  the  test  candidate  file  MIS  has  main  effects,  no  quadratic  effects,  and  one 
two-way  interaction  (WAR__FHRS  x  RBS_RDGOAL).  Also,  BAT  and  IWO  test 
candidate  files  both  have  main  effects,  one  quadratic  effect  (RBS_RDGOAL  x 
RBS_RDGOAL),  and  one  two-way  interaction  (WAR_FHRS  x  RBS_RDGOAL). 
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2.  Meta-model  Statistics  Using  Stepwise  Regression 

To  further  compare  the  prediction  power  between  both  HST  Full  and  Reduced 
Meta-models,  the  percent  error  for  each  is  displayed  in  Figure  17  (x-axis  in  percent).  The 
category  Meta-model  (Full)  includes  those  models  developed  using  stepwise  regression, 
but  with  low  t  Ratios  remaining.  However,  the  Meta-model  (Reduced)  comprises  the 
models  with  t  Ratios  that  have  an  absolute  value  of  ten  or  greater  (low  magnitude  t  Ratios 
removed).  The  red  box  plot  is  the  prediction  error  for  the  reduced  meta-model  and  the 
blue  box  plot  represents  the  prediction  errors  in  the  full  meta-model.  Significantly,  both 
full  and  reduced  meta-models  have  50%  of  their  predictions  of  NAVARM  RBS  cost 
within  the  0.05%  to  2%  error  range.  More  importantly,  it  shows  the  similarity  of  both  full 
and  reduced  meta- models.  In  addition,  the  nine  other  test  candidate  percent  error  box 
plots  are  available  in  Appendix  E.  The  results  of  those  nine  test  candidate  percent  error 
box  plots  display  for  all  cases  that  nearly  75%  of  their  predictions  of  NAVARM  RBS 
cost  are  less  than  3%  of  error. 

The  meta-model  statistics  are  available  in  Table  3  for  NAVARM  RBS  cost.  The 
table  contains  the  statistical  measures  of  the  meta-models  available  in  Appendix  D.  The 
significance  of  Table  3  is  to  illustrate  that  the  removal  of  the  low  end  t  Ratio  factor  does 
not  drastically  change  the  performance  of  the  meta-model.  In  fact,  there  are  some  test 
candidates  that  experience  minor  changes  in  R-square  adjusted  and  RMSE,  but  nearly 
double  in  value  for  the  F  Ratio  as  effects  are  removed  from  the  meta-models. 
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Percent  Error  (Full  Meta-model)  &  Percent  Error  (Reduced  Meta-model) 
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Note:  The  red  box  plot  is  HST  reduced  meta-model.  The  blue  box  plot  is  the  full  meta¬ 
model.  The  x-axis  is  percent  error  calculated  by  the  difference  between  actual  and 
estimated,  divided  by  actual. 

Figure  17.  HST  RBS  cost  prediction  error  for  full  and  reduced  meta-models 


Table  3.  Test  candidate  meta-model  statistics  for  NAY  ARM  RBS  cost 


RBS  Total  Cost  Meta-model  Statistical  Table 

Meta-model  (Full) 

Meta-model  (Reduced 

Rsquare  Adjusted 

RMSE 

F  Ratio 

Rsquare  Adjusted 

RMSE 

F  Ratio 

HST 

0.98 

1408638 

1499 

0.98 

1649191 

2656 

LEM 

0.98 

1092954 

901 

0.96 

1395977 

1631 

BAT 

0.97 

3463740 

1003 

0.96 

4558055 

1199 

BON 

0.98 

1850437 

1300 

0.97 

2432405 

2066 

NOR 

0.93 

421271 

631 

0.93 

441221 

1256 

MAL 

0.97 

3628082 

626 

0.95 

4492968 

1159 

rvvo 

0.97 

3130217 

679 

0.95 

3977324 

1031 

MIS 

0.98 

724969 

1181 

0.97 

897656 

2287 

OCA 

0.98 

1532184 

725 

0.96 

1895954 

1505 

DEN 

0.98 

1638973 

1379 

0.98 

1821543 

2498 
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3. 


Influential  Factors  Results 


Finally,  we  provide  the  NAVARM  RBS  cost  factor  influence  ranking.  In  Table  4, 
a  list  of  factors  from  left  to  right  is  displayed  for  each  candidate  file.  The  list  is  gathered 
from  their  meta-model  developed  in  stepwise  regression.  The  ranking  of  the  factors  is  1 
to  13,  representing  largest  to  smallest  magnitudes  for  the  t  Ratios,  respectively.  Factors  in 
red  text  are  the  main  effects  that  are  statistically  significant,  but  have  been  removed  from 
the  model  due  to  the  t  Ratio  being  smaller  than  ten  (i.e.,  not  practically  significant). 

Additionally,  we  count  how  many  times  each  practically  significant  factor  appears 
in  all  test  problems.  We  find  that  the  overall  most  influential  factors  on  cost  are  (in  order 
of  importance):  goal,  Unit  Price,  wartime  flying  hours,  maintenance  rate  to  failure, 

high  priority  order  and  ship  time,  number  of  aircraft,  wholesale  delay  time,  ratable  pool 
factor,  intermediate  maintenance  activity  repair  time,  and  mean  time  to  repair. 

All  the  test  cases,  except  MAL,  have  either  goal  or  unit  priceas  their  number 
one  factor.  The  MAL  test  case  has  wartime  flying  hours  as  its  number  one  factor  with 
unit  priceas  second,  and  goal  as  its  third.  As  mentioned  in  Chapter  I,  Section  A,  the 
Marine  Corps  is  operating  with  less  than  half  their  aircraft  available.  This  suggests  that 
the  remaining  aircraft  are  being  overused,  resulting  in  greater  wear  and  tear  and  yielding 
reduced  airworthiness.  Since  this  is  based  on  retrospective  data  we  cannot  establish 
causality,  but  further  investigation  seems  indicated. 
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Table  4.  NAV ARM’s  RBS  cost  influence  to  factors  by  t  ratio  ranking 


RBS  Total  Cost  Factor  Influence  by  t  Ratio  Rankin 

’from  Meta-models 

1 

2 

3 

4 

5 

6 

1 

8 

9 

10 

11 

12 

13 

HST 

RBSEDGOAL 

IMPRICE 

WARFHRS 

MET 

HPOST 

WSnumber 

WHSLDELAY 

EPF 

IMARPRTM 

mm 

LEM 

RBSEDGOAL 

WARJMS 

I11TPEICE 

EPF 

IMARPRTM 

MRF 

HP 0ST 

WS  number 

WHSLDELAY 

QPA 

BAT 

RBSEDGOAL 

IMPRICE 

WARMS 

MEF 

HP 0ST 

WSmimber 

mm 

EPF 

QPA 

WHSLDELAY 

IMARPRTM 

BOX 

RBSEDGOAL 

IMPRICE 

WARMS 

MEF 

WHSL  DELAY 

HP 0ST 

nm 

WSnumber 

EPF 

IMARPRTM 

NOR 

IMPRICE 

RBSEDGOAL 

WARFHRS 

MEF 

HP 0ST 

QPA 

■ 

UAL 

WARFHRS 

IMPRICE 

RBS  RDGOAL 

MEF 

RPF 

IMA  RPR  TM 

WS  number 

WHSLDELAY 

HPOST 

m 

iwo 

RBSEDGOAL 

IMPRICE 

WARMS 

MEF 

WSjmnber 

HP 0ST 

WHSLDELAY 

Mm 

RPF 

QPA 

ins 

RBSEDGOAL 

IMPRICE 

MRF 

WARFHRS 

HPOST 

WSmimber 

WHSLDELAY 

Mm 

RPF 

IMARPRTM 

QPA 

OCA 

IMPRICE 

RB$RDG0A1 

WARMS 

MEF 

HP 0ST 

IMARPRTM 

WHSLDELAY 

EPF 

WSjmber 

DEN 

RBSEDGOAL 

UMPRICE 

WARMS 

MEF 

WHSL  DELAY 

HPOST 

WS  number 

D.  NAVSUP  TOOL 

After  identifying  the  NAV ARM  output  sensitivities  and  developing  the  meta¬ 
models,  an  estimation  tool  was  developed  for  NAVSUP  WSS  in  Excel  using  VBA.  The 
tool  affords  NAVSUP  WSS,  Office  Code  N421,  the  ability  to  make  adjustments  to 
multiple  factors  simultaneously,  and  see  how  that  affects  NAV ARM  RBS  cost. 
Implementation  of  NED  will  aid  N421  in  training  and  planning,  and  will  improve  their 
overall  understanding  of  factors  that  affect  RBS  sensitivity. 
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V.  CONCLUSION 


We  demonstrate  that  the  most  influential  factors  to  NAVARM  RBS  cost  are 
availability  goal,  Unit  Price,  wartime  flying  hours,  maintenance  rate  to  failure,  high 
priority  order  and  ship  time,  number  of  aircraft,  wholesale  delay  time,  ratable  pool  factor, 
intermediate  maintenance  activity  repair  time,  and  mean  time  to  repair. 

Prior  to  this  research.  Sax  (2012,  Appendix  1-4)  discovered  that  wholesale  delay 
time  and  high  priority  order  and  ship  time  were  the  drivers  behind  the  RBS  model.  This 
thesis  found  that  both  factors  influence  the  output,  but  they  are  not  the  most  influential. 
We  suggest  that  future  work  consider  a  DOE  that  varies  both  factors  as  continuous 
integers  rather  than  scaling  from  the  baseline  value. 

In  conducting  the  OAT  design  and  assessing  the  scatter  plots  analysis,  we  note 
that  these  historical  methods  cannot  reliably  determine  which  factors  are  most  influential, 
nor  can  they  provide  accurate  estimates  of  RBS  cost.  Stepwise  regression,  by  contrast, 
succeeds  at  both.  Our  findings  are  that  60%  of  the  models  have  only  main  effects  (no 
two-way  interactions  or  quadratic  effects).  However,  four  test  candidate  files  have  a 
quadratic  effect  (RBS_RDGOAL  x  RBS_RDGOAL).  The  test  candidate  files  with  the 
RBS_RDGOAL  quadratic  effect  are  USS  Bataan  (LHD  5),  USS  BonHomme  Richard 
(LHD  6),  USS  Iwo  Jima  (LHD  7),  and  FMS  Denmark.  These  four  test  candidate  files  are 
for  sites  with  rotary  wing  aircraft  parts,  but  we  cannot  conclude  that  rotary  wing  aircraft 
cause  this  effect.  The  MAL  test  case  has  unique  factor  ranking,  and  suggests  further  study 
in  order  to  explain  these  differences. 

NED  is  developed  as  a  predictive  tool  for  NAVARM  RBS  cost  based  on  the 
stepwise  regression  models  for  the  ten  test  cases,  and  produces  predictions  of  cost  when 
factors  vary  within  the  scaled  range.  The  NED  meta-model  for  the  USS  Harry  S.  Truman 
has  50%  of  its  predictions  within  the  0.05%  to  2%  error  range.  The  results  of  the  other 
nine  test  candidate  files  have  nearly  75%  of  their  predictions  within  a  3%  or  less  error 
while  predicting  RBS  cost,  and  NED  allows  the  user  to  make  predictions  of  cost  for  all 
test  cases  within  7%  of  actual. 
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Simulation  distinguishes  between  verification  and  validation — the  former 
corresponds  to  debugging  the  model  while  the  latter  corresponds  to  assessing  model 
correctness.  A  large-scale  space-filling  design  such  as  the  NOB  acts  a  stress  test  on 
simulation  models,  often  exposing  software  bugs  and  vulnerabilities.  The  NOB  cannot 
establish  validity,  but  the  fact  that  NAVARM  was  able  to  successfully  run  all  input 
configurations  generated  by  the  design  lends  credence  to  it  as  a  well- verified  model. 

Another  potential  direction  for  future  development  is  to  pool  all  ten  test  cases  to 
see  whether  a  single  comprehensive  meta-model  can  be  constructed.  This  would  allow 
investigation  of  possible  model  commonalities  across  the  scenarios. 

Lastly,  a  future  study  should  consider  different  ranges  of  scaling  than  were  used 
in  the  current  work.  This  could  change  the  sensitives  of  the  response  to  the  various 
factors  as  well  as  the  degree  of  non-linearity  or  interaction  effects. 
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APPENDIX  A.  512  -  POINT  NOB  DOE  FACTOR  CORRELATION 
AND  SCATTERPLOT  MATRIX 


Correlations 


MKf 

RPf 


WS 


Note:  Correlation  and  scatterplot  matrix  show  that  NOB  DOE  is  space  filling  with  no  correlation. 
This  makes  for  an  excellent  way  to  experiment  with  multiple  factors  covering  their  full  spectrum. 
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Total  Cost  (M)  Total  Cost  (M) 


APPENDIX  B.  OAT  SA  GRAPHS  OF  MIS/BON/OCA/BAT 


A.  OAT  DESIGN  RESULTS  FOR  MIS 


MIS  OAT  Sensitivity  Analysis  of  RBS  Cost 
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OAT  DESIGN  RESULTS  FOR  BON 


BON  OAT  Sensitivity  Analysis  of  RBS  Cost 
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Total  Cost  (M)  Total  Cost  (M) 


C 


OAT  DESIGN  RESULTS  FOR  OCA 


OCA  OAT  Sensitivity  Analysis  of  RBS  Cost 

$300 
$200 
$100 
$0 

0.010.100.200.300.400.500.600.700.800.901.001.101.201.301.401.501.601.701.801.902.00 

Percent  Change  from  Baseline 

QPA  IMA_RPR_TM  EXP_PRG_W  HP_OST 

WHSL_DELAY^— UNITPRICE  MRF  RPF 

FLY_HRS  MTTR  WAR_FHRS  WS_number 

RBS_RDGOAL 

OAT  DESIGN  RESULTS  FOR  BAT 


BAT  OAT  Sensitivity  Analysis  of  RBS  Cost 
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APPENDIX  C.  HST  FACTOR  BY  OUTPUT  SCATTER  PLOTS 


~  Bivariate  Fit  of  RBS  Total  Cost  By  QPA 

130000000  . 


i  -  Bivariate  Fit  of  RBS  Total  Cost  By  IMA  RPR  TM 

130000000 


0.9  0.92  0.94  0.96  0.98  1  1.02  1.04  1.06  1.08  1.1 

QPA 

w - Linear  Fit 

A  Linear  Fit 

RBS  Total  Cost  =  84464353  +  945605.98*QPA 

.  i  Summary  of  Fit 

RSquare  0.000023 

RSquareAdj  -0.00194 

Root  Mean  Square  Error  11409161 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  5 1 2 

- 1  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  1.532e+12  1.532e+12  0.0118 

Error  510  6.6386e+16  1.302e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.9137 

■  i  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  84464353  8730914  9.67  <.0001* 

QPA  945605.98  8716342  0.11  0.9137 


0.9  0.92  0.94  0.96  0.98  1  1.02  1.04  1.06  1.08  1.1 

IMA.RPR.TM 

w - Linear  Fit 

^  Linear  Fit 

RBS  Total  Cost  -  72189208  +  13220751*IMA_RPR_TM 

1  Summary  of  Fit 

RSquare  0.004511 

RSquareAdj  0.002559 

Root  Mean  Square  Error  11383531 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  5 1 2 

i  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  2.9947e+14  2.995e+14  2.3110 

Error  510  6.6088e+16  1.296e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.1291 

1  Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  72189208  8711300  8.29  <.0001* 

IMA_RPR_TM  13220751  8696761  1.52  0.1291 


a.  QPA  by  RBS  cost 


b.  IMA_RPR_TM  by  RBS  cost 


-  Bivariate  Fit  of  RBS  Total  Cost  By  EXP_PRG_W 
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A  ▼  Bivariate  Fit  of  RBS  Total  Cost  By  HP_OST 

1 30000000 


120000000 


0.9  0.92  0.94  0.96  0.98  1  1.02  1.04  1.06  1.08  1.1 

HP_OST 


w - Linear  Fit  | 

J  Linear  Fit 

RBS  Total  Cost  =  8661 5193  - 1 205234.8*EXP_PRG_W 


A  Summary  of  Fit 


RSquare 

3.749e-5 

RSquareAdj 

-0.00192 

Root  Mean  Square  Error 

11409079 

Mean  of  Response 

85409959 

Observations  (or  Sum  Wgts) 

512 

A  Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

2.4888e+12 

2.489e+12 

0.0191 

Error 

510 

6.6385e+16 

1.302e+14 

Prob  >  F 

C.  Total 

511 

6.6388e+16 

0.8901 

A  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  86615193  8730851  9.92  <.0001* 

EXP_PRG_W  -1205235  8716279  -0.14  0.8901 


^——Linear  Fit 

A  Linear  Fit 

RBS  Total  Cost  =  46075191  +  39334767*HP_OST 


A  Summary  of  Fit 

RSquare  0.03993 

RSquareAdj  0.038048 

Root  Mean  Square  Error  11179183 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 

^  Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

1 

2.6509e+15 

2.651e+15 

21.2115 

Error 

510 

6.3737e+16 

1.25e+14 

Prob  >  F 

C.  Total 

511 

6.6388e+16 

<.0001* 

A  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  46075191  8554922  5.39  <.0001* 
HP_OST  39334767  8540644  4.61  <.0001* 


a.  EXP_PRG_W  by  RBS  cost 


b.  HP_OST  by  RBS  cost 
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RBS  Total  Cost 


-  Bivariate  Fit  of  RBS  Total  Cost  By  WHSL_DELAY 

130000000 


41  -  Bivariate  Fit  of  RBS  Total  Cost  By  UNITPRICE 
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— —  Linear  Fit 

4  Linear  Fit 


RBS  Total  Cost  =  54539840  +  308701 19*WHSL_DELAY 


4  Summary  of  Fit 

RSquare 
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RSquare  Adj 

0.022681 

Root  Mean  Square  Error 

11268120 

Mean  of  Response 

85409959 

Observations  (or  Sum  Wgts) 
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Analysis  of  Variance 

Sum  of 


Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  1.6327e+15  1.633e+15  12.8592 

Error  510  6.4755e+16  1.27e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.0004* 

4l  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  54539840  8622981  6.32  <.0001* 

WHSL_DELAY  30870119  8608590  3.59  0.0004* 


— —Linear  Fit 

4  Linear  Fit 

RBS  Total  Cost  =  -2788655  +  88198614*UNITPRICE 


41  Summary  of  Fit 


RSquare 
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RSquare  Adj 
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Root  Mean  Square  Error 
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Mean  of  Response 
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Observations  (or  Sum  Wgts) 

512 

4  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  1.3328e+16  1.333e+16  128.1053 

Error  510  5.306e+16  1.04e+14  Prob  >  F 

C.  Total  511  6.6388e+16  <.0001* 

<1  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  -2788655  7805552  -0.36  0.7210 

UNITPRICE  88198614  7792525  11.32  <.0001* 


a.  WHSL_DELAY  by  RBS  cost 


b.  UNITPRICE  by  RBS  cost 


▼  Bivariate  Fit  of  RBS  Total  Cost  By  FLY_HRS 
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- Linear  Fit  | 

41  Linear  Fit 

RBS  Total  Cost  =  84538438  +  871 521 .1 6*FLY_HRS 


41  Summary  of  Fit 

RSquare  1.96e-5 

RSquare  Adj  -0.001 94 

Root  Mean  Square  Error  11409181 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 

4  Analysis  of  Variance 


Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  1.3014e+12  1.301e+12  0.0100 

Error  510  6.6386e+16  1.302e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.9204 

<1  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  84538438  8730929  9.68  <.0001* 

FLY_HRS  871521.16  8716357  0.10  0.9204 


~ - Linear  Fit  | 

41  Linear  Fit 

RBS  Total  Cost  =  81 288448  +4121511 ,2*MTTR 


4  Summary  of  Fit 

RSquare  0.000438 

RSquare  Adj  -0.00152 

Root  Mean  Square  Error  1 1406792 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 

4  Analysis  of  Variance 


Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  2.9104e+13  2.91  e+ 13  0.2237 

Error  510  6.6359e+16  1.301e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.6365 

41  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  81288448  8729101  9.31  <.0001* 

MTTR  4121511.2  8714532  0.47  0.6365 


a.  FLY_HRS  by  RBS  cost 


b.  MTTR  by  RBS  cost 
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RBS  Total  Cost 


-  Bivariate  Fit  of  RBS  Total  Cost  By  WAR_FHRS 
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A  ▼  Bivariate  Fit  of  RBS  Total  Cost  By  WS  number 
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—Linear  Fit 

A  Linear  Fit 

RBS  Total  Cost  =  1431 195.7  +  83978763*WAR_FHRS 

A  Summary  of  Fit 

RSquare  0.182008 

RSquare  Adj  0.180404 

Root  Mean  Square  Error  1 0318897 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 

A  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  1.2083e+16  1.208e+16  113.4779 

Error  510  5.4305e+16  1.065e+14  Prob  >  F 

C.  Total  511  6.6388e+16  <.0001* 

A  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  1431195.7  7896584  0.18  0.8562 

WAR_FHRS  83978763  7883405  10.65  <.0001* 


w - Linear  Fit 

A  Linear  Fit 

RBS  Total  Cost  =  1 1 80301 64  -  32620205*WS_number 


Summary  of  Fit 

RSquare  0.027462 

RSquare  Adj  0.025555 

Root  Mean  Square  Error  11251 544 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 


-  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  1.8231e+15  1.823e+15  14.4008 

Error  510  6.4565e+16  1.266e+14  Prob  >  F 

C.  Total  511  6.6388e+16  0.0002* 

Parameter  Estimates 

Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  118030164  8610297  13.71  <.0001* 

WS_number  -32620205  8595926  -3.79  0.0002* 


a.  WAR_FHRS  by  RBS  cost 


b.  WS_number  by  RBS  cost 


-  Bivariate  Fit  of  RBS  Total  Cost  By  RBS  RDGOAL 

130000000 

120000000 
110000000 
O  100000000 
1 

£  90000000 

*  80000000 
70000000 
60000000 

0^9  0.92  0.94  0.96  0.98  1  1.02  1.04  1.06  1.08  l!l 

RBS.RDGOAL 

w - Linear  Fit 

■  i  Linear  Fit 

RBS  Total  Cost  =  -34589187  +  1 19999145*RBS.RDGOAL 


.1  Summary  of  Fit 

RSquare  0.371627 

RSquare  Adj  0.370395 

Root  Mean  Square  Error  9044143 

Mean  of  Response  85409959 

Observations  (or  Sum  Wgts)  512 


'  Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  1  2.4671  e+ 16  2.467e+16  301.6202 

Error  510  4.1716e+16  8.18e+13  Prob  >  F 

C.  Total  511  6.6388e+16  <.0001* 

A  Parameter  Estimates 
Term  Estimate  Std  Error  t  Ratio  Prob>|t| 

Intercept  -34589187  6921072  -5.00  <.0001* 

RBS_RDGOAL  119999145  6909521  17.37  <.0001* 


a.  RBS_RDGOAL  by  RBS  cost 
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APPENDIX  D.  TEST  CANDIDATE  FILE  META-MODELS 


A.  LEM  RBS  REGRESSION  ANALYSIS 


Meta-model  (Full) 

Actual  by  Predicted  Plot 


35000000  45000000  55000000  65000000  75000000 
RES  Tool  Cost  Predicted  RMSE:i  le«6RSq;0.96 
PVifc*<.0001 

i  Summary  of  Fit 

RSquare  0.977973 

RSquare  Adj  0.976887 

Root  Mean  Square  Error  1 092954 

Mean  of  Response  50590103 

Observations  (or  Sum  Wgts)  5 1 2 

Analysis  of  Variance 


Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

24 

2.5828e*16 

1.076e-15 

900.9058 

Error 

487 

5.8174e-14 

1.195e*12 

Prob  >  F 

C,  Total 

511 

2.641e-16 

Studentized  Residuals 


0  50  100  ISO  200  250  300  350  400  450  500 


Row  Number 


Meta-model  (Reduced) 


Actual  by  Predicted  Plot 

75000000 

•  / 

70000000 

•  •/ 

•  K 

2  65000000 

|  60000000 

u  55000000 

8  50000000 

j»  45000000  i| 

pp  * 

40300000 

35000000 

35000000  450000005500000065000000 

RBS  Total  Cost  Predicted  RMSEs  1  4e-6  RSqs0.96 

PValue<0001 

Summary  of  Fit 

RSquare 

0.962884 

RSquare  Adj 

0.962294 

Root  Mean  Square  Error 

1395977 

Mean  of  Response 

50590103 

Observations  (or  Sum  Wgts) 

512 

Analysis  of  Variance 


Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  8  2.543e*16  3.179e*15  1631.153 

Error  503  9.8022e*14  1.949e-12  Prob  >  F 

C  Total  511  2.641e-16 


Studentized  Residuals 

5 


0  50  100  ISO  200  250  300  350  400  450  500 

Rnw 


Sorted  Parameter  Estimates 


Term  btinufe  Std  Error  t  Ratio  Pro6>|tJ 


RSS.ROGOAt 

WAR  FhRS 

UNfiPfiJCI 

68797509  8442615 
54440687  8416585 
52732352  8420533 
33416111  844843.1 
32034809  8440*89 
30964751  653496 

22481000  8437438 
-20775080  8428541 

8149 

6466 

6262 

3955 

3755 

3628 

2664 

-2465 

«25r 

W 

IMA  RPR  TM 

MR*' 

HP.OST 

WS.nyfflbe^ 

r~n  <ooot* 

i~l  <0001* 

Si  I  <0001* 

■ 

MRF-1)*(WAR_FHRS-1) 

88555450  14919574 

594 

iMA.RPR.TM-1  )*I  RSS.ROGOAl-1)  -8-45^0655  1  52  305TB 

-5.55 

UN7TPR<£-1)-,RSS.RaCA'-i) 

7382120*  15004922 

492 

WSl.OCtAY 

39048447  8455973 

462 

WS  .nurOff-IjMtSS.  ROGOAl-1) 

-66857656  1  5071608 

•444 

uwPfta*ir<wAR  fHRs-i) 

64875172  14838464 

4J7 

K>F-1)*!RSS.ROGOaI-1) 

-63506187  14531401 

■4  37 

WP_OS7-1)*(WAR_FhRS-T) 

59250914  14573069 

4.07 

UA  RPR  TM-1)*  WAR  FMRS-1) 

54861415  14519532 

378 

fi?F  - 1  )*(WAR_FhRS-  1) 

52426474  14624794 

358 

00004* 

WAR_FmR$-1)*{WAR_FhRS-1) 

55921933  16365083 

342 

00007’ 

•WS  number-1  )*(W$  number- 1) 

52635435  164S05S3 

320 

SPA 

24191987  8556994 

283 

00049* 

WWl  OFtAY-IJVWAR  FhRS-T) 

37981119  14948886 

2-54 

00114* 

iMA  RPR  TI4-1  r  fUNfTPRJCE- 1 ) 

36870307  14763032 

2.50 
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BAT  RBS  REGRESSION  ANALYSIS 


Meta-model  (Full) 


Meta-model  (Reduced) 


Actual  by  Predicted  Plot 

200000000 
T  90000000 

-  laoooocoo 

3  170000000 

i  1 60 000000 

$  150000000 
s  140000000 
&  130000000 

i  120000000 
110000000 
100000000 
90000000 

100000000  130000000  16OOO0WO  19-0000000 
RBS  local  Cost  Predicted  «M5E=3.5e*6  BSq=0.97 
PValue  *  ,QG0 1 


Actual  by  Predicted  Plot 


R&SToUl  tort  Prc dnUri  fl*A$£  =4.fln:-6  ftSqaO  W 
PWucc.ftOOi 


Summary  of  Fit 

RSquare 

0.974035 

RSquaneAdj 

0970004 

Root  Mean  Square  Error 

5465740 

Mean  of  Response 

1  i364e*B 

Observations  (or  Sum  Vtffts) 

SI  l 

Analysis  of  Variance 

Sum  of 


Source 

DF 

Square! 

Mean  Square 

F  Ratio 

Model 

IS 

2-2666*17 

1 .203*16 

1005.123 

Error 

492 

5.9020c*  15 

12*13 

Prob>  F 

C  Total 

511 

2.5457^17 

<.0001" 

:  Summary  of  Fit 

RSquam  0.955537 

RSquareAdj  0.95474 

Rgqe  Wean  Square  £rw  4553055 

Mean  of  Rtipense  1.564**0 

Observation*  (or  Sum  WgttJ  51 1 

Analysis  of  Variance 


Sum  of 


Source 

DF 

Square! 

Mean  Square 

FRatw 

Model 

9  22414*17 

2.44*16 

1196  70S 

Error 

502 

14429*16 

2.076*13 

Prob  >  F 

C  Total 

511 

22457*17 

*.0001  ■ 

HewNym&ftf 


Sorted  Parameter  Estimates 

Term 


Estimate  Std' Error  tftefo 


RBS  flCKMA L  5107KJW2  31113® 

UNrTPRJCE  liKGHtt  2fi6«3fl1 

mjm  123153431  2WJ1Z0 

MRF  9Q2523W  2661521 

(RES JKOGOfll-I^RBS  RDGQAL-1J  UHM  S51923W 

HP_0ST  4S1S9102  2672114 

ffi  nimbff  -434DCW69  2650705 

MTTR  50702530  26512T5 

m  2720S4Z1  2671596 

(WAR,  FH  HS-1 )'!  &.BS  RDGCAl- 1 1  470050531  4W76341 

110.52 

52.72 

4^26 

3J« 

im 

1l.» 

-'fl.37 

11.» 

hub 

9.5Q 

1 

,  '  '  '  '  1 

1  <0001- 
<0001“ 
<-0001- 
<0001' 
<mv 
<0001- 
<.0001' 

<0001' 

3 

i 

i 

] 

Pi  i  1 

i 

4 4m  11 

■a-.ii 

UHPRJCE-1),(RK.RDG(M-1] 

126736877  49240576 

6.84 

■:  ODor 

IRA. 

176157M  2601224 

6-57 

*  0301“ 

WHHJKLAIf 

1664961S  2678531 

5.22 

<  QUO  1 ‘ 

m  RPRTW 

14959090  2659271 

W 

*  0301 ’ 

IMTTiR.I  n0Bi.RSXSOAJ.-l3 

24001444^  4S0TO170 

5-13 

'RPF-ITCRBS  RDGOAH) 

18441111  41768290 

3.7B 

imimiwhohai-i) 

15015S24S  49S4VH4 

3.23 

0.0013- 

WH5L D£LAY'l],fflflS RDQOflLj  1) 

•128*1  4956S257 

■3.» 

Ptavb>[i| 
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BON  RBS  REGRESSION  ANALYSIS 


Meta-model  (Full) 

Act 0*1  by  Predicted  Plot 


MS  Tout  Co*  PfoOKUtf  MISf  •  t  »*•«  tt*«0  99 
0001 


Meta-model  (Reduced) 

Act 0*1  by  Predicted  Plot 


MS  Total  Celt  fcefrctad  RMSE.2  U-6  RSq.O  97 
RV*A»e<  0001 


Summary  of  Fit 

RSquare  09*5272 

RSquare  A^)  0  9*4514 

Root  MeanSqu**  Em*  1*50437 

Mean  of  Response  1.149e*8 

Observe*™  (or  Sum  Wgti)  512 

Analysis  of  Variance 


Source 

Of 

Sum  of 
Square! 

Mean  Square 

f  Ratio 

Model 

2J  i.m2«*ir 

4453e«l5 

1300  46* 

Error 

4*6 

1.6641e*15 

3424e-12 

Prob  >  F 

C  Total 

511 

1.1299e*17 

Summary  of  Fit 

RSqu*re 

0973713 

RSquare  Adj 

0-973242 

Root  Mean  Square  Error 

2432405 

Mean  of  Response 

U49e<* 

Observations  (or  Sum  Wgts] 

512 

Analysis  of  Variance 

Sum  of 


Source 

Of 

Squares 

Mean  Square 

f  Ratio 

Modd 

9 

1.10O2e*17 

1  222e*16 

206*090 

Error 

502 

29701e-15 

5917e.12 

Prob  >  F 

C.  Total 

511 

1.1299e*17 

<0001* 

Studentued  fteuduafc 


0  »  W  DO  »  MO  IS  «0  4S  M 

bilumr 


Studenttted  Rnduah 

,  I 


d  u.-* ayt  .  .s.  fojL 


K  too  no  MO  i)0  NO  IS  400  4»  MO 


Sorted  Parameter  Estimates 


Term 

Estimate 

Std  Error 

t  Ratio 

Prob>  |t| 

RBS.ROGOAL 

169*39609 

1441351 

11743 

1  « 4001* 

UWTPfUCE 

117624790 

1440302 

*167 

‘ _ 

<0001* 

WARJHRS 

93*7265* 

1441042 

65.14 

, .0001* 

MRf 

81205221 

1449723 

56  01 

<  0001* 

WHSl.OCLAV 

39310111 

1441156 

2726 

•  0001* 

MP.OST 

ttMMI 

1444781 

2242 

a 

•0001* 

l RBS.ROGOAL- 1)* (RBS.ROGOAL- 1)  493419063 

27952192 

1765 

■ 

<0001* 

MTTR 

13949550 

1446657 

964 

i 

<0001* 

VS^um&e< 

1354*607 

1435902 

-9  44 

-UfTWa  - 1  PRBS.RDGOAt-1 

iuim 

fWAR.f  MRS- 1  p  R3S.RDGOAI- 1 ) 

19923422* 

24*54576 

*02 

!  -  0001' 

RPF 

11167076 

1440441 

7.75 

,  -0001' 

iMRf-l)-(RBS.RDGOAL-l) 

173209950 

24*05399 

6  96 

•  0001- 

IMA  RPR.TM 

66775119 

143*073 

464 

l  •  000 1  * 

(RPf-l)*(R8S.RDGOAL-1) 

97053740 

246**345 

393 

I  •  0001* 

(UNtTPRCE- 1  )*  (WAR.f  MRS- 1 ) 

91067892 

25238910 

361 

0  0001* 

(f  L  V.HRS- 1  P'RSS.RDGOAl  - 1) 

MM 09M 

2M  83781 

343 

D  OX  *  ’ 

(MTTR- 1  )*(WS.number- 1) 

-83*91 714 

25803560 

-325 

0  0012* 

rws.numbee- 1  )•( RBS.ROGOAL  - 1 ) 

•82139689 

25629024 

-320 

lMRF-irCWAR.fHRS-1) 

71*81707 

2562502* 

241 

lWHSl.0CLAV-1)TWAR.fHRS-1) 

70163*7* 

254213*7 

276 

1  0  0060* 

I  RBS.ROGOAL- 1  p  Demands- 1) 

6*6*5372 

24939533 

2.75 

0  0061  • 

(UWTPRlCE-inMRM) 

69375945 

25341096 

2.74 

:  0  0064* 

iULtiS 

-11 04943 

1410103 

-029 

OT42 
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NOR  RBS  REGRESSION  ANALYSIS 


Meta-model  (Fall) 

Actual  by  Predicted  Plot 


240 00000 
23QQQ0Q0 
|  2200O0OQ 
<  21 000000 
|  20000000 
g  19000000 
^  laoooooo 
“  17000000 
10000000 
15000000 


1 6000000 1 BOOOOOO  2OODQO0O  22000000 
RBS  Total  Cost  Predicted  RMSE±421271  ftfq^Ci-93 


PVaU*<.£K?tH 


Meta-model  (Reduced) 

Actual  by  Predicted!  Plot 


■2 

£ 

< 

3 


■  ■■ 

« 


2400-0000 
23000000 
22000000 
21000000 
2000-3000 
1 9000000' 
1 8000000 
17000000 
1 6000000 
1 5000000' 


1 60*00000 1 SOOOOOO  2QOQODOO  2  2000000' 

RES  Total  "Cost  Predicted  RMSE=441221  RSq=0.B3 
P  Value'-. 000  T 


3  Summary  of  Fit 

RSqusre  0-932814 

RSquare  Adj  0.931336 

Root  M  ean  Square  Error  42 1 271,1 

M  ear  of  Respon se  1 8S0  5  769 

Ofceivati  ons  (or  5u  m  Wgts)  5 1!  2 

1  Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

11 

1<232e+l5 

l,12e+14 

631-0923 

Emor 

500 

8.6735e-13 

1,775e+H1 

Prob  >  F 

C  Total 

511 

1.3207e-15 

<0001* 

Summary  of  Fit 

RSquare  0.92  S41 6 

RSquartAdj  0924679 

Root  Mean  Square  Error  44 1 2  206 

Mean  of  Response  1iS805  769 

Observations  {or  Sum  Wgts)  5 1 2 

Analysis  of  Variance 

Sum  of 


Source 

OF 

Squares 

Mean  Square 

F  Ratio 

Model 

5 

1.2222e+15 

2,444e-l4 

1255  652 

Error 

506 

9.8506^13 

1.947^11 

Prob  >  F 

C  Total 

511 

U207e-15 

<0001* 

StudenTtod  R«  idua  h 


0  m  too  iso  :o&  :w  j»  im  *o&  mb- 

ImIMv 


Student  izcd  Residuals 


l 

S  ijf . 

L£ 
h 


a  5D  TOO  15D  2  DO  250  JDO  350  *D0  450  500 

ftsw  Humber 


Sorted  Parameter  Estimates 

Term  Estimate  S(d  Error  t  Ratio 

UNITPRICE  19010967  323S9&4  5669 

RBS  RDGOAL  10546531  3236562  32.59 

WARJFHRS  9661943.6  32S1I2J  29-72 

MRF  83963813  3226664  2603 

HP QST_ 7619163-5  3247K6  2346 


mm  'fio&j 

QPA  1126546,5  3229001  3.49 

[MRf-1}*£RBS  RDGGA1-1J  17900854  5569015  3.21 

[H  PjOST- 1  J'(MRF-I)  17216669  5613170  3.07 

[UNITPEUCE'  1  )*  (WAfl^FHRS-  1 )  16143146  S6B4S16  2.64 

hf  cBi- 1  unuprice- 1 )  14377211  5616919  3,56 1 


Prob>  |t| 
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MAL  RBS  REGRESSION  ANALYSIS 


Meta-model  (Full) 

Actual  by  Predicted  Plot 


Meta-model  (Reduced) 

Actual  by  Predicted  Plot 


ttS  Total  Cost  Prcdded  RMSE«3  6e*6  RSqaO  97 
PValue<  0001 


240000000 

220400000 


8 . 

^  1*0000000 

g  1*0004000 

140040000 


124000000 


’20000000  1*0000000  200000000  240000000 

m  Total  Com  4MSI  >4  U-l  ^ .0  9) 

iva^oooi 


Summary  of  Fit 


Summary  of  Fit 


RSouare 

0.971066 

RSquare  Adj 

0969515 

Root  Mean  Square  Error 

3628062 

Mean  of  Response 

1.763e-8 

Observations  (or  Sum  Wgts) 

512 

Analysis  of  Variance 

Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  26  2.U26e-17  8241e-15  626.0473 

Error  485  6384^15  1.316e-13  Prob>F 

C  Total  511  2.2064e*17 


Studentind  Residuab 


0  »  104  150  240  2»  M0  J»  400  4»  500 


Sob  iMto 


RSquare  0.954071 

RSquare  Adj  0.953248 

Root  Mean  Square  Error  4492968 

Mean  of  Response  1 .763e*8 

Observabons  (or  Sum  Wgts)  512 

Analysis  of  Variance 


Source 

DF 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

9  2.1051e*17 

2.339e*16 

1158.665 

Error 

502 

1.0134e*16 

2.019e*13 

Prob  >  F 

C  Total 

511 

2.2064 e*17 

Studcnttxcd  Residuals 


0  54  100  ISO  200  230  100  ISO  400  4S0  500 

fta»N*<4ar 


Sorted  Parameter  Estimates 


Term 

htimate 

Std  Error 

t  Ratio 

Prob>|t| 

WAR  FMRS 

207700717 

2841509 

73-10 

- 1 

UNITPRia 

179824747 

2804042 

6413 

\ - 1 

R8S.RDGOAL 

33093671 

6392589 

51.77 

MRF 

2851774 

3660 

RPF 

96986322 

2807665 

3454 

IMA.RPR.TM 

85044390 

2831884 

3003 

WS  number 

*75655175 

2803672 

•2696 

*0001* 

WHSl.DaAY 

48612758 

2823396 

1722 

HP  Q5T 

Smt 

22203496 

2812992 

Hi 

WAR  FHRS-1)-{RSS  ROGOAl-O-95) 

876318702 

1  16e«8 

7.55 

<0001* 

RPF-1)*(R8S_RDGOAL-0  9S) 

715476780 

1 I14e*8 

6.42 

1MA.RPR.TM- 1  )*(RE$.RDGOAl-0  95) 

591832500 

1.117e*S 

5.30 

!MA_ftPR_TM- 1  )‘(WAR_FHRS- 1) 

242351787 

48729206 

4.97 

1 

R8S.RDGOAL-0.95r.RBS.RDGOAl-0«)  1.1472e*9 

2  366*-* 

485 

j 

UNrTPRlCE-1)*(MRM) 

230734269 

49572153 

465 

1 

UN(TPRJCE-1)*/R3S  RDGOAl-095) 

478512803 

1.112e*8 

4.30 

[  j 

<  0001* 

WS.number*  1  )*(R8S.RDGOAL-0  95) 

-4493e*8 

1.118  e*8 

*4.02 

1 

<.0001* 

[UNlTPRlCE- 1  nWAR.FHRS*  1 ) 

190320656 

49090583 

388 

1 

00001* 

QPA-1)*(WHSJXUY-T) 

-1.762e-»8 

49597450 

*355 

1 

00004* 

"WAR  FHRS-1)*(WAR  FHRS-1) 

161933691 

55121412 

2  94 

00035* 

RPF-i)-(WAR_FHRS-i) 

144415578 

49594593 

291 

"WAR.FHRS- 1  )*(WS.number- 1 ) 

-1363e-8 

49504666 

-2.75 

1 

IMA  RPRJM.1)*S'UWTPR1CM) 

123707102 

49128817 

2.52 

00121* 

iMA.RPRlTM-l)'(RPf-l) 

124494452 

49689705 

2.51 
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IWO  RBS  REGRESSION  ANALYSIS 


Meta-model  (Full) 

Actual  by  Predicted  Plot 

200030000 
190000000 
180003000 
170003000 
l  163003000 
£  150000000 
|  140000000 
£  130003000 
“  120003000 
110003000 
100003000 

110300000  140000000  170000000 

RBS  Total  Cost  Predicted  RMS*  >  3. 1  e-6  RSqi0.97 
PValue<OOOl 


Meta-model  (Reduced) 

Actual  by  Predicted  Plot 

200000000 
190000000 
«  180000000 
|  170000000 
5  160000000 
^  150000000 
O  140000000 
«  130000000 
120000000 
110000000 
100000000 

110030000  140000000  170000000 
m  Total  Co*  Predicted  RM$*:4*»6  RSq=0  95 
PValue<  0001 


Summary  of  Fit 

RSquare  0.972151 

RSquare  Adj  0  970718 

Root  Mean  Square  Error  81 30217 
Mean  of  Response  1  398e*8 

Observations  (or  Sum  Wgts)  5 1 2 

Analysis  of  Variance 


Sum  of 

Source  Of  Squares  Mean  Square  f  Ratio 

Model  25  1.66238+17  6.649e-15  678.5996 

Error  486  4  762e*15  9.798e+12  Prob>F 

C  Total  511  1.7099e*17 


'  Summary  of  Fit 

RSquare 

0.95865 

RSquare  Adj 

0.952725 

Root  Mean  Square  Error 

3977324 

Mean  of  Response 

1.398e-8 

Observations  (or  Sum  Wgts) 

512 

i  Analysis  of  Variance 

Sum  of 


Source 

DF 

Squares 

Mean  Square 

F  Ratio 

Model 

10 

1.6306e-17 

1.631e-16 

1030-804 

Error 

501 

7.9254e*15 

1.582e*13 

Prob  >  F 

C  Total 

511 

1.7099e*17 

<.0001* 

Student  tied  Reirduah 


few***** 


Studentkzed  Rtsiduab 


0  50  100  ISO  200  250  300  350  400  430  S00 

Bow  Number 


Sorted  Parameter  Estimates 


Term 

lUknate 

Std  fr  roe 

«  Ratio 

Proto >  (t| 

R8S.ROGOAI 

27)000570 

2923659 

9)38 

1  <0001* 

UNfTPRJCI 

145570512 

2413765 

6031 

<0001* 

WAR.FHRS 

121961800 

2435287 

50  08 

•  0001* 

MRf 

97693809 

2450502 

3987 

-0001* 

(RaS.ROGOAi.inRtS.RDGOAll)  1  3662*. 9 

62472485 

2187 

•.0001* 

WS.ftgmbe 

*48728083 

2412727 

-2020 

•0001* 

MP.OST 

4.  809008 

24400)1 

17.42 

<.0001* 

WW$t_0ClAV 

34019642 

2440661 

1394 

•.0001* 

MTTR 

34096827 

2448595 

1393 

9 

•  0001* 

0  14 

•J  W"e-I 

l 

Pf 

17295662 

2447294 

707 

1 

•  0001* 

MTTR- 1  )*(R8S„RDGOAt  •  1 ) 

326598949 

48925069 

668 

•  0001* 

LfNfTPR»a  •  1  >*<  RtS.ROGOAl  - 1 ) 

119879670 

5141)030 

610 

•  0001* 

MRf  •  1  }*(R8S.ROGOAl  •  1) 

282499900 

5070437) 

518 

|  •  0001* 

JPA 

11253855 

2442245 

461 

j  •  000 1* 

MRf  *  1  )*(WS_numto#r- 1) 

*1  802e*8 

42850234 

•421 

•  0001* 

jNtTptua .  i  nwAR_r  mrs*  i  ) 

154809956 

4255)062 

364 

0  0003* 

HNfTPR>CM)*<MTTR-1) 

150953856 

41666713 

362 

o  oxn* 

WAR.FHRS- 1  1 ) 

•1.519e-« 

43016669 

•353 

oooev 

QPAinMTTRI) 

131122755 

42344163 

3  14 

00018' 

URf  •  1  )*(WAR_FmRS*1) 

126682688 

42613742 

297 

0  00)  1  • 

VlA.RPR.TVt 

69735242 

2406572 

290 

0  00)9* 

MRF.ir(MTTR.I) 

117499298 

40923754 

287 

0  004 1  * 

RPf-1)*(RBS.ROGOAL.1) 

140878184 

50401614 
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MIS  RBS  REGRESSION  ANALYSIS 


Meta-model  (F  ull) 


Actual  by  Predicted  Plot 


RBS  Total  Cost  Predicted  RM$C=724969  RSq=0  98 
Pv»lue<000l 

Summary  of  Fit 

RSquare  0.983102 

RSquare  Adj  0.982269 

Root  Mean  Square  Error  7249683 

Mean  of  Response  49240315 

Observations  (or  Sum  Wgts)  5 1 2 

Analysis  of  Variance 


Source 

Of 

Sum  of 
Squares 

Mean  Square 

F  Ratio 

Model 

24 

1 .4891  e-16 

6.205e-U 

1180.538 

Error 

487 

2.5596e*14 

5.256e*11 

Prob  >  F 

C  Total 

511 

1.5147e*16 

Meta-model  (Reduced) 

Actual  by  Predicted  Plot 

70000000 

65000000 

j  60000000 

o  55000000 
|  50000000 
*  45000000 
40000000 

40000000  50000000  60000000 

R8S  Total  Cost  Predicted  RMSE  =897656  RSq=0  97 
PVa»we<0001 


Summary  of  Fit 

RScfwe  0.973242 

RSquare  Adj  0.972816 

Root  Mean  Square  Error  89765S9 

Mean  of  Response  49240315 

Observations  (or  Sum  Wgts)  512 


Analysis  of  Variance 

Sum  of 


Source 

OF 

Squares 

Mean  Square 

F  Ratio 

Model 

8  U742e-16 

1  843e*l5 

2286872 

Error 

503 

4.0531e-14 

8058e-11 

Prob  >  F 

C  Total 

511 

1.5147e*16 

<0001* 

Studentizcd  Residuals 


Studcntued  Residuals 


1 


■ a  V  - . ■ 

0  SO  tOO  ISO  240  260  106  JS0  400  4 »  100 

fo«  Numb* 


o  m  too  tw  joo  *»  )»  no  4oo  4%  mo 

low  Numb*# 


Sorted  Parameter  Estimates 

Term  Estimate  Std  trror  t  Ratio  P»ob>|t| 


RBS  RDGOAl 

UNITPRKE 

MRf 

WARwFHRS 

HP.OST 

WS.numbee 

WHSl.DELAY 

(WAR  FHRS-mRBS  RDGOAL-1) 

51606742 

49694952 

36044619 

33572104 

227B1453 

-12161783 

87479082 

5613067 

5596923 

565483-5 

5589528 

5621662 

5560142 

5626368 

91 J 94 
88.79 
6374 
6006 
4032 
*2188 
1555 
9.71 

k -  1 

B  i 

4TZ9004S 

5619979 

7.35 

f  UNfTPRlCE-  1  )*(R8S_RDGOAL- 1 ) 

69325140 

10012185 

6.92 

RPF 

30144951 

5665523 

5.32 

IMA  RPR  TM 

265617&1 

5586642 

4.75 

QPA 

2328510.7 

5695442 

4.09 

<  ooor 

(WS  number- 1)*; RBS  RDGOAl-1) 

•39867773 

10026002 

•398 

1  RBS.RDGO  Ai  -in  RBS.RDGO  Ai  - 1 } 

4:o:-es?9 

10880402 

385 

( MRF  - 1  )*(  WS.number- 1 ) 

*37869973 

9964089 

*380 

1 

(WAR_FhRS-  1  ),(WS.number- 1 ) 

*36993153 

9885263 

-374 

0  0002 

(MRMnWAR.fHRS-1) 

36165387 

9943034 

364 

*i  i-»\-  • 

IQPA-I)'(MTTR-I) 

33621269 

9846426 

341 

(UNfTPRlCE- 1  )*(MRF- 1 ) 

33456922 

9909621 

3.36 

(UNfTPRlCE*  1  }*(WAR.FH  RS- 1 ) 

32637117 

9832229 

332 

fWHSl  DELAY*  l)\'WS.number-1) 

-32184120 

9866793 

-326 

1 

(QPA-1)*(WHSL.D6LAY*1) 

•32178961 

9958210 

•3.23 

(HP  OST-1)‘(WAR^FMRS*1| 

26508347 

9671862 

2.74 

1 
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OCA  RBS  REGRESSION  ANALYSIS 


Meta-model  (Full) 

Actual  by  Predicted  Plot 


Meta-model  (Reduced) 

Actual  by  Predicted  Plot 


R85  Tout  C<»t  Predicted  RMSE*1  $e»6  RSq.098 


RIS  Total  Cost  »rt*c*d  RM5E  s  t  9e*6  R$q«0  96 


Summary  of  Fit 

Rvalue-  0001 

I  Summary  of  Fit 

PV**<0001 

RSquare 

0.977588 

RSquare 

0.964258 

RSquare  Adj 

0.976239 

RSquare  Adj 

0.963617 

Root  Mean  Square  Error 

1532184 

Root  Mean  Square  Error 

1895954 

Mean  of  Response 

1.159e~8 

Mean  of  Response 

1.159e*8 

Observations  (or  Sum  Wgts) 

512 

Observations  (or  Sum  Wgts) 

512 

Analysis  of  Variance 


Analysis  of  Variance 


Sum  of 

Source  Df  Squares  Mean  Square  f  Ratio 

Model  29  4.9356e*16  1.702e-15  724.9679 

Error  4*2  1.1315e-15  2.3 48*-12  Prob  >  F 

C  Total  511  5.0487e-16 

Student  ixed  Residuals 


-44 - 1 - .  ■  i  ■  i  ■  i - 1 - ■  . 

9  59  tOO  ISO  200  2»  MO  150  *X  4»  500 


Sum  of 

Source  DF  Squares  Mean  Square  F  Ratio 

Model  9  4  8683e*16  5.409e-15  1504.799 

Error  502  1.8045e*15  3.595e*12  Prob  >  F 

C.  Total  511  5.0487e*16 

Studcntizcd  Residuals 


•««* 


Sorted  Parameter  Estimates 

Term  Estimate  Std  Error  I  Ratio  Prot»|t| 


UNfTPRtCE 

RBS.RDGOAl 

WAR.FHRS 

MR F 

MP.OST 

IMA  RPR_TM 

WHSL_D€lAV 

RPf 

118991962 

74775187 

49471032 

49858738 

27863390 

23026387 

22749541 

21332433 

1192590 

1196701 

1186757 

1202449 

1193234 

1190749 

1193701 

1194134 

1165907 

9978 

6248 

4169 

4146 

2335 

1934 

1906 

1786 

-ooor 

“3  *  0001* 

5J3 

fWAR_FHRS-1)*(R8S  RDC-OAl-l) 

112628759 

20742023 

543 

«  0001* 

(WAR  FHRS-1)*(WAR  FHRS-1) 

113200793 

23043205 

491 

-0001* 

(MRf-1)-iR8S.RDGOAl.T) 

99081343 

20792897 

4,77 

-0001* 

(HP.O$T-1)’(R8S.ROGOAL- 1 ) 

95797312 

20690525 

463 

fWAR.FH  RS- 1 ) *  (WS^rvumtoer- 1 ) 

-97261768 

21066789 

-4  62 

«.ooov 

(HP.OST- 1  )*(WAR_FHRS-1) 

79306618 

20473788 

347 

0  0001* 

(MRF-1)*(WAR  FHRS-1) 

78641461 

20926617 

376 

(RPM)-(R8S_RiXOAl-1) 

76584507 

20399516 

375 

0  0002* 

(WMSL.D6LAV.  1  )*(WAR  JH  RS-i ) 

77049150 

21031633 

366 

0  0003* 

(WS.number- 1  )*(WS.ovrn6er- 1 ) 

82614809 

22942240 

361 

00003* 

(hp  ost-ihmrf-i) 

73414366 

20538991 

3  57 

0  0004* 

(UNfTPRlCE- 1  )*(RSSJtDGOAl-1 ) 

69956565 

21124814 

3.31 

0.0010* 

(IMA  RPR  TM-1)*(RPf-1) 

67498444 

20936592 

3  22 

0  0014* 

(WHSL.DflAV.  1  )*  (UNtTPIUCE.  1) 

64286993 

20556826 

313 

00019* 

(WS.number- 1  )*{RBS_RDGCAi- 1 ) 

-61843536 

21234303 

-291 

0  0088* 

MM  A  RPR_TM-1)*(WAR_FHRS-1) 

57826899 

20419922 

283 

0  0048* 

(UNfTPRlCE-1)*(MRF-1) 

58134011 

21012633 

277 

0  0059* 

( 1MA_RPR_TM- 1  )*(R8S  RDGOAl-T) 

58948326 

21331321 

2.76 

0  0059* 
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DEN  RBS  REGRESSION  ANALYSIS 


Meta-model  (F  ull) 


Actual  by  Predicted  Plot 


RfiS  Tout  Cott  Predict'd  RMSf=1  6«*6  RSq=0  96 
PVHue<0001 


Meta-model  (Reduced) 

Actual  by  Predicted  Plot 


RBS  Tout  Cost  Predated  RMSC  ■  1  8t *6  RSq«0.98 
PValue<0001 


Summary  of  Fit 

RSquare  0  980518 

RSquare  Adj  0979607 

Root  Mean  Square  Error  1 638973 

Mean  of  Response  752 1 6261 

Observations  (or  Sum  Wgts)  5 1 2 


'Analysis  of  Variance 

Sum  of 

Source  Df  Squares  Mean  Square  f  Ratio 

Model  18  6.6654e*16  3,703e*15  1  378.502 

Error  493  1.3243e-15  2.686e*12  Prob>F 

C  Total  511  6.7978e*16  <0001* 


Student  tied  Residuals 


o  so  too  iso  :oo  :so  ko  no  roo  4  so  wo 


Raw!**-**’ 


Summary  of  Fit 

RSquare  0.975448 

RSquare  Adj  0.975058 

Root  Mean  Square  Error  1 82 1 543 

Mean  of  Response  7521  6261 

Observations  (or  Sum  Wgts)  512 


Analysis  of  Variance 

Sum  of 

Source  Df  Squares  Mean  Square  f  Ratio 

Model  8  6.6309e+16  8.289e*15  2498064 

Error  503  1.669e-15  3.318e-12  Prob>F 

C.  Total  511  6.7978e*16 


Student  ned  Residuals 


0  SO  100  ISO  200  290  900  330  400  430  300 


«to»*Nu*nb«r 


Sorted  Parameter  Estimates 


fern 

Estimate  Std  Error 

t  Ratio 

Prob>  |t| 

R8S.RDGOAI 

140779143  1268823 

11095 

<0001 

UNJTPWCE 

78005821  1265679 

6163 

<0001* 

WARJHRS 

65918743  1275799 

51.67 

<  ooor 

MRf 

64289693  1269165 

50  66 

<0001* 

WHSl.C&AY 

39170067  1260933 

3106 

•0001* 

HP.OST 

23174409  1267074 

1829 

<0001* 

lR8S.ROGOAl.1)*(R8$.ROGOAt.1)  426483385  245338(8 

1738 

1 

<0001* 

W$numb<r 

-21229495  1258605 

•1687 

i  <0001* 

WHSi.OClAY*!  )*(MRf -1) 

85195364  21665967 

393 

*  '»}1* 

VNlTPRtCMnWAR  fHRS-1) 

83631789  22221761 

376 

:  Xu 

WAft.F  HRS*  1  )*(R8S_R0GQAl  *  1 ) 

78787654  21801368 

361 

:  o  ocr 

HP.OS7-1)*(MTTR*1) 

-78793428  22774336 

-346 

.  J  j>- 

MRf-1)'(WAR  FHRS-1) 

71120002  22078089 

322 

:  oou- 

MRf-1)*(R8S_ROGOAl-1) 

64920129  21811159 

298 

:•  oca  r 

UWTPRJCE-1  f*(MRF-1) 

58819667  22248865 

264 

j>D6V 

MRf-1)*(MTTR«1) 

55267739  21347489 

259 

-um 
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APPENDIX  E.  FULL  AND  REDUCED  META-MODEL  ERROR 


A.  LEM  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 


B.  BAT  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 
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BON  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 


D.  NOR  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 
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MAL  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 


Percent  Error  (Full  Meta-model)  &  Percent  Error  (Reduced  Meta-model) 


Group  X 


Color 


Size 


-o  Percent  Error  (Full  Meta-model) 

-O  Percent  Error  (Reduced  Meta-modd) 


Percent  Error  (Full  Meta-modd)  &  Percent  Error  (Reduced  Meta-modd) 


IWO  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 


Percent  Error  (Full  Meta-model)  &  Percent  Error  (Reduced  Meta-model) 


67 


G 


MIS  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 


H.  OCA  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 


Percent  Error  (Full  Meta-model)  &  Percent  Error  (Reduced  Meta-model) 
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L  DEN  FULL  VERSUS  REDUCED  META-MODEL  PREDICTION  ERROR 
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